Release of version 4.0.5

Motivated by #140.

docs/_templates/navbar.html

@@ -50,6 +50,8 @@
                   <li><a href="https://github.com/instaloader/instaloader">Instaloader on GitHub</a></li>
                   <li role="separator" class="divider"></li>
                   <li style="padding-left: 20px;"><a class="github-button" href="https://github.com/instaloader/instaloader" data-icon="octicon-star" data-size="large" data-show-count="true" aria-label="Star Instaloader on GitHub">Star</a></li>
+                  <li role="separator" class="divider"></li>
+                  <li><a href="https://www.paypal.me/aandergr"><b>Donate (via Paypal)</b></a></li>
                 </ul>
               </li>
             </ul>

docs/basic-usage.rst

@@ -68,7 +68,7 @@ Instaloader supports the following targets:
    **download the user's stories**.
 
 - ``"#hashtag"``
-   Posts with a certain **hashtag** (the quotes are usually neccessary),
+   Posts with a certain **hashtag** (the quotes are usually necessary),
 
 - ``:stories``
    The currently-visible **stories** of your followees (requires
@@ -158,7 +158,7 @@ Filter Posts
 .. py:currentmodule:: instaloader
 
 The options :option:`--post-filter` and :option:`--storyitem-filter`
-allows to specify criterias that posts or story items have to
+allows to specify criteria that posts or story items have to
 meet to be downloaded. If not given, all posts are downloaded.
 
 The filter string must be a

docs/codesnippets.rst

@@ -11,8 +11,8 @@ Advanced Instaloader Examples
    :backlinks: none
 
 Here we present code examples that use the :ref:`python-module-instaloader` for
-more advanced tasks than what is possible with the Instaloader command line
+more advanced Instagram downloading or metadata mining than what is possible
-interface.
+with the Instaloader command line interface.
 
 .. For each code snippet:
    - title
@@ -21,12 +21,12 @@ interface.
    - link to discussion issue
    - link used methods
 
-Download Posts in a specific period
+Download Posts in a Specific Period
 -----------------------------------
 
-To collect pictures (and metadata) only from a specific period, you can play
+To only download Instagram pictures (and metadata) that are within a specific
-around with :func:`~itertools.dropwhile` and :func:`~itertools.takewhile` from
+period, you can play around with :func:`~itertools.dropwhile` and
-:mod:`itertools` like in this snippet.
+:func:`~itertools.takewhile` from :mod:`itertools` like in this snippet.
 
 .. literalinclude:: codesnippets/121_since_until.py
 
@@ -37,8 +37,8 @@ Discussed in :issue:`121`.
 Likes of a Profile / Ghost Followers
 ------------------------------------
 
-To store inactive followers, i.e. followers that did not like any of your
+To obtain a list of your inactive followers, i.e. followers that did not like
-pictures, into a file you can use this approach.
+any of your pictures, into a file you can use this approach.
 
 .. literalinclude:: codesnippets/120_ghost_followers.py
 
@@ -51,10 +51,10 @@ Discussed in :issue:`120`.
 Track Deleted Posts
 -------------------
 
-This script uses Instaloader to obtain a list of currently-online posts, and
+This script uses Instaloader to obtain a list of currently-online Instagram and
-generates the matching filename of each post. It outputs a list of posts which
+compares it with the set of posts that you already have downloaded.  It outputs
-are online but not offline (i.e. not yet downloaded) and a list of posts which
+a list of posts which are online but not offline (i.e. not yet downloaded) and a
-are offline but not online (i.e. deleted in the profile).
+list of posts which are offline but not online (i.e. deleted in the profile).
 
 .. literalinclude:: codesnippets/56_track_deleted.py
 
@@ -68,9 +68,9 @@ Only one Post per User
 
 To download only the one most recent post from each user, this snippet creates a
 :class:`set` that contains the users of which a post has already been
-downloaded. When iterating the posts, check whether the post's owner already is
+downloaded. While iterating the posts, it checks whether the post's owner
-in the set. If so, skip the post. Otherwise, download it and add the user to
+already is in the set. If not, the post is downloaded from Instagram and the
-that set.
+user is added to that set.
 
 .. literalinclude:: codesnippets/113_only_one_per_user.py
 
@@ -79,23 +79,23 @@ See also :class:`Post`, :meth:`Instaloader.download_post`,
 
 Discussed in :issue:`113`.
 
-Upgrade Images by local Copies
+Upgrade Images by Local Copies
 ------------------------------
 
 The following script finds local versions of images fetched by Instaloader, in
-order to ugprade the downloaded images by locally-found versions with better
+order to upgrade the downloaded images by locally-found versions with better
 quality. It uses image hashing to identify similar images.
 
 `updgrade-instaloader-images.py <https://gist.github.com/pavelkryukov/15f93d19a99428a284a8bcec27e0187b>`__ (external link to GitHub Gist)
 
 Discussed in :issue:`46`.
 
-Render Captions to downloaded Images
+Add Captions to Images
-------------------------------------
+----------------------
 
 Instaloader does not modify the downloaded JPEG file. However, one could combine
-it with an imaging library such as Pillow or PIL to render the
+it with an imaging library such as Pillow or PIL to render the caption on
-:attr:`Post.caption` on pictures.  The following shows an approach.
+Instagram pictures.  The following shows an approach.
 
 .. literalinclude:: codesnippets/110_pil_captions.py
 
@@ -103,3 +103,26 @@ See also :attr:`Post.caption`, :attr:`Post.url`, :meth:`Post.from_shortcode`,
 :func:`load_structure_from_file`.
 
 Discussed in :issue:`110`.
+
+Metadata JSON Files
+-------------------
+
+The JSON files Instaloader saves along with each Post contain all the metadata
+that has been retrieved from Instagram while downloading the picture and
+associated required information.
+
+With `jq <https://stedolan.github.io/jq/>`__, a command-line JSON processor, the
+metadata can be easily post-processed. For example, Instaloader's JSON files can
+be pretty-formatted with:
+
+.. code-block:: none
+
+   xzcat 2018-05-13_11-18-45_UTC.json.xz | jq .node
+
+However, Instaloader tries to do as few metadata requests as possible, so,
+depending on how Instaloader has been invoked, it may occur that these files do
+not contain the complete available metadata structure. Nevertheless, the file
+can be loaded into Instaloader with :func:`load_structure_from_file` and the
+required metadata then be accessed via the :class:`Post` or :class:`Profile`
+attributes, which trigger an Instagram request if that particular information is
+not present in the JSON file.

docs/contributing.rst

@@ -1,3 +1,5 @@
+.. _contributing:
+
 Contributing to Instaloader
 ===========================
 
@@ -23,10 +25,10 @@ following in mind:
 
 - Briefly **check whether the bug has already been reported**. If you find an
   issue reporting the same bug you encountered, comment there rather than
-  opening a new issue. However, if unsure, it is advised to create a new issue.
+  opening a new issue. However, if unsure, please create a new issue.
 
 - State **how the bug can be reproduced**, i.e. how Instaloader was invoked
-  when the problem occured (of course, you may anonymize profile names etc.).
+  when the problem occurred (of course, you may anonymize profile names etc.).
 
 - Include all **error messages and tracebacks** in the report.
 
@@ -47,13 +49,13 @@ few things to consider:
   you forked from the most recent branch.
 
 - We use `Pylint <https://www.pylint.org/>`__ for error and syntax checking of
-  ``instaloader.py``. The file ``.travis.yml`` in the project's root directory
+  the source. The file ``.travis.yml`` in the project's root directory
   shows how it is invoked. Note that sometimes it might be better to disable a
   warning rather than adapting the code to Pylint's desires.
 
 - The documentation source is located in the ``docs`` folder. The file
-  ``cli-options.rst`` is merely an RST-formatted copy of ``ìnstaloader --help``
+  ``cli-options.rst`` is merely an RST-formatted copy of ``instaloader --help``
-  output, of which the source is near the bottom of ``instaloader.py``.
+  output, of which the source is in ``instaloader/__main__.py``.
 
 - Feel free to contact us, even if you "only" have Proof-of-Concepts or
   not-fully integrated changes. They already might be an advance for the
@@ -81,3 +83,21 @@ Please consider the following:
 - **Be patient**. Naturally, bugs and pull requests have a higher priority than
   feature suggestions. Keep in mind that this is a free software project, and
   unfortunately we only have limited time to work on it.
+
+Donations
+---------
+
+.. donations-start
+
+It is a pleasure for us to share our Instaloader to the world, and we are proud
+to have attracted such an active and motivating community, with so many users
+who share their suggestions and ideas with us. Buying a community-sponsored beer
+or coffee from time to time is very likely to further raise our passion for the
+development of Instaloader.
+
+We have a PayPal.Me link for donations: `PayPal.me/aandergr
+<https://www.paypal.me/aandergr>`__
+
+.. donations-end
+
+.. (Discussion in :issue:`130`)

docs/index.rst

@@ -70,10 +70,23 @@ Useful Links
 - `Issue Tracker / Bug Tracker <https://github.com/instaloader/instaloader/issues>`__
 - `Version History <https://github.com/instaloader/instaloader/releases>`__
 
+Disclaimer
+----------
+
 .. include:: ../README.rst
    :start-after: disclaimer-start
    :end-before: disclaimer-end
 
+Contributing
+------------
+
+As an open source project, Instaloader heavily depends on the contributions from
+its community. See :ref:`contributing` for how you may help Instaloader to
+become an even greater tool.
+
+.. include:: contributing.rst
+   :start-after: donations-start
+   :end-before: donations-end
 
 ..
    * :ref:`genindex`

instaloader/__init__.py

@@ -1,7 +1,7 @@
 """Download pictures (or videos) along with their captions and other metadata from Instagram."""
 
 
-__version__ = '4.0.4'
+__version__ = '4.0.5'
 
 
 try:

instaloader/instaloader.py

@@ -270,14 +270,15 @@ class Instaloader:
         with self.context.get_anonymous_session() as anonymous_session:
             date_object = datetime.strptime(anonymous_session.head(profile_pic_url).headers["Last-Modified"],
                                             '%a, %d %b %Y %H:%M:%S GMT')
+        profile_pic_extension = 'jpg'
         if ((format_string_contains_key(self.dirname_pattern, 'profile') or
              format_string_contains_key(self.dirname_pattern, 'target'))):
             filename = '{0}/{1}_UTC_profile_pic.{2}'.format(self.dirname_pattern.format(profile=profile.username.lower(),
                                                                                         target=profile.username.lower()),
-                                                            _epoch_to_string(date_object), profile_pic_url[-3:])
+                                                            _epoch_to_string(date_object), profile_pic_extension)
         else:
             filename = '{0}/{1}_{2}_UTC_profile_pic.{3}'.format(self.dirname_pattern.format(), profile.username.lower(),
-                                                                _epoch_to_string(date_object), profile_pic_url[-3:])
+                                                                _epoch_to_string(date_object), profile_pic_extension)
         if os.path.isfile(filename):
             self.context.log(filename + ' already exists')
             return None

instaloader/structures.py

@@ -93,10 +93,9 @@ class Post:
         return b64encode(mediaid.to_bytes(9, 'big'), b'-_').decode().replace('A', ' ').lstrip().replace(' ', 'A')
 
     def _asdict(self):
+        node = self._node
         if self._full_metadata_dict:
-            node = self._full_metadata_dict
+            node.update(self._full_metadata_dict)
-        else:
-            node = self._node
         if self._owner_profile:
             node['owner'] = self.owner_profile._asdict()
         if self._location:

test/instaloader_unittests.py

@@ -40,16 +40,21 @@ class TestInstaloaderAnonymously(unittest.TestCase):
         print("Removing {}".format(self.dir))
         shutil.rmtree(self.dir)
 
+    def post_paging_test(self, iterator):
+        previous_post = None
+        for post in islice(iterator, PAGING_MAX_COUNT):
+            print(post)
+            if previous_post:
+                self.assertTrue(post.date_utc < previous_post.date_utc)
+            previous_post = post
+
     @unittest.SkipTest
     def test_public_profile_download(self):
         self.L.download_profile(PUBLIC_PROFILE, profile_pic=False, fast_update=True)
         self.L.download_profile(PUBLIC_PROFILE, profile_pic=False, fast_update=True)
 
     def test_public_profile_paging(self):
-        for count, post in enumerate(instaloader.Profile.from_username(self.L.context, PUBLIC_PROFILE).get_posts()):
+        self.post_paging_test(instaloader.Profile.from_username(self.L.context, PUBLIC_PROFILE).get_posts())
-            print(post)
-            if count == PAGING_MAX_COUNT:
-                break
 
     def test_profile_pic_download(self):
         self.L.download_profile(PUBLIC_PROFILE, profile_pic_only=True)
@@ -93,10 +98,7 @@ class TestInstaloaderLoggedIn(TestInstaloaderAnonymously):
                 print(item)
 
     def test_private_profile_paging(self):
-        for count, post in enumerate(instaloader.Profile.from_username(self.L.context, PRIVATE_PROFILE).get_posts()):
+        self.post_paging_test(instaloader.Profile.from_username(self.L.context, PRIVATE_PROFILE).get_posts())
-            print(post)
-            if count == PAGING_MAX_COUNT:
-                break
 
     def test_feed_download(self):
         self.L.download_feed_posts(NORMAL_MAX_COUNT)