Release of version 4.2.1

Download HD profile picture if logged in
Unfortunately, it is now required to be logged in to access the HD version of profile pictures. When attempting to download profile pictures without --login, a warning message is printed once, and the lower-quality versions are obtained. For backwards compatibility, already-downloaded profile pictures are overwritten if the now-obtainable version is assumed to be of better quality than the existing one (determined by file size vs Content-Length). The iPhone endpoint is accessed with code exhumed from c355338010. Also, this reverts "Profile: don't access removed iphone info endpoint" 08327c4117. This fixes #209.
2026-10-17 06:02:18 -07:00 · 2019-01-17 17:47:45 +01:00 · 2019-01-17 17:39:29 +01:00 · 2019-01-17 11:39:18 +01:00 · 2019-01-17 11:00:04 +01:00 · 2019-01-08 19:38:46 +01:00
31 changed files with 868 additions and 224 deletions
@@ -8,7 +8,7 @@ install:
 - pip install pylint requests
 - pip install -r docs/requirements.txt
 script:
- - python3 -m pylint -r n -d bad-whitespace,missing-docstring,too-many-arguments,locally-disabled,line-too-long,too-many-public-methods,too-many-lines,too-many-instance-attributes,too-many-locals,too-many-branches,too-many-statements,inconsistent-return-statements,invalid-name,wildcard-import,unused-wildcard-import,no-else-return instaloader
+ - python3 -m pylint -r n -d bad-whitespace,missing-docstring,too-many-arguments,locally-disabled,line-too-long,too-many-public-methods,too-many-lines,too-many-instance-attributes,too-many-locals,too-many-branches,too-many-statements,inconsistent-return-statements,invalid-name,wildcard-import,unused-wildcard-import,no-else-return,cyclic-import,unnecessary-pass instaloader
 - make -C docs html SPHINXOPTS="-W -n"
 deploy:
  - provider: pypi
@@ -0,0 +1,75 @@
+@import url(bootstrap-4.1.3.min.css);
+a {
+  color: #008d06; }
+  a:hover, a.current, a.active {
+    color: #f48400; }
+  a.current, a.active {
+    font-weight: 500; }
+
+.doc-sidebar a:hover, .modal-body a:hover {
+  text-decoration: none; }
+
+code {
+  color: #212529; }
+
+.highlight {
+  border-radius: 0.3em; }
+  .highlight pre {
+    padding: 0.7em;
+    color: #fff; }
+  .highlight .c1 {
+    color: #008d06; }
+  .highlight .k, .highlight .kn, .highlight .ow {
+    color: #f48400;
+    font-weight: bold; }
+  .highlight .nb {
+    color: #f48400; }
+  .highlight .s1, .highlight .s2, .highlight .mi {
+    color: #ff7efd; }
+
+.doc-content {
+  padding-top: 0.5em;
+  top: 3.5625em;
+  position: fixed;
+  bottom: 0;
+  overflow-y: auto; }
+
+.doc-sidebar {
+  position: fixed;
+  padding-top: 0.5em;
+  top: 3.5625em;
+  bottom: 0;
+  right: 0;
+  overflow-y: auto; }
+  .doc-sidebar ul {
+    list-style-type: none;
+    padding-left: 1em; }
+    .doc-sidebar ul.current {
+      padding-left: 0; }
+  .doc-sidebar li {
+    padding-top: 0.2em;
+    padding-bottom: 0.2em; }
+
+.admonition {
+  border-radius: 0.3em;
+  border: 1px solid #ccc;
+  background-color: #f8f9fa; }
+  .admonition .admonition-title {
+    color: #ff7efd; }
+
+.topic {
+  border-radius: 0.3em;
+  background-color: #f8f9fa; }
+
+.nav-link {
+  padding-top: 0;
+  padding-bottom: 0; }
+
+.btn-success {
+  background-color: #008d06;
+  border-color: #008d06; }
+  .btn-success:hover {
+    background-color: #f48400;
+    border-color: #f48400; }
+
+/*# sourceMappingURL=instaloader.css.map */
@@ -0,0 +1,22 @@
+$(function () {
+    $('[data-toggle="tooltip"]').tooltip();
+
+    $('.doc-sidebar > ul > li.current').attr("id", "localtoc");
+    $('#localtoc ul').addClass("nav flex-column");
+    $('#localtoc ul li').addClass("nav-item");
+    $('#localtoc ul li a').addClass("nav-link");
+    $('.doc-content').scrollspy({target: '#localtoc'});
+
+    const top_href = '#' + $('.section:first').attr("id");
+    $('#localtoc > a.current').attr("href", top_href);
+
+    $('#navbarToc a').on("click", function () {
+        const href = $(this).attr("href");
+        if (href === '#') {
+            window.location.href = top_href;
+        } else {
+            window.location.href = href;
+        }
+        $('#navbarToc').modal('hide');
+    });
+});
@@ -0,0 +1,117 @@
+@import 'bootstrap-4.1.3.min.css';
+
+$color_instaloader_main: #008d06;
+$color_instaloader_accent: #f48400;
+$color_instaloader_hyper: #ff7efd;
+$color_normal_text: #212529;
+$color_light_bg: #f8f9fa;
+
+a {
+  color: $color_instaloader_main;
+
+  &:hover, &.current, &.active {
+    color: $color_instaloader_accent;
+  }
+
+  &.current, &.active {
+    font-weight: 500;
+  }
+}
+
+.doc-sidebar, .modal-body {
+  a:hover {
+    text-decoration: none
+  }
+}
+
+code {
+  color: $color_normal_text;
+}
+
+.highlight {
+  border-radius: 0.3em;
+
+  pre {
+    padding: 0.7em;
+    color: #fff;
+  }
+
+  .c1 {
+    color: $color_instaloader_main
+  }
+
+  .k, .kn, .ow {
+    color: $color_instaloader_accent;
+    font-weight: bold
+  }
+
+  .nb {
+    color: $color_instaloader_accent
+  }
+
+  .s1, .s2, .mi {
+    color: $color_instaloader_hyper
+  }
+}
+
+.doc-content {
+  padding-top: 0.5em;
+  top: 3.5625em;
+  position: fixed;
+  bottom: 0;
+  overflow-y: auto;
+}
+
+.doc-sidebar {
+  position: fixed;
+  padding-top: 0.5em;
+  top: 3.5625em;
+  bottom: 0;
+  right: 0;
+  overflow-y: auto;
+
+  ul {
+    list-style-type: none;
+    padding-left: 1em;
+
+    &.current {
+      padding-left: 0;
+    }
+  }
+
+  li {
+    padding-top: 0.2em;
+    padding-bottom: 0.2em;
+
+  }
+}
+
+.admonition {
+  border-radius: 0.3em;
+  border: 1px solid #ccc;
+  background-color: $color_light_bg;
+
+  .admonition-title {
+    color: $color_instaloader_hyper;
+  }
+}
+
+.topic {
+  border-radius: 0.3em;
+  background-color: $color_light_bg;
+}
+
+.nav-link {
+  padding-top: 0;
+  padding-bottom: 0;
+}
+
+.btn-success {
+  background-color: $color_instaloader_main;
+  border-color: $color_instaloader_main;
+
+  &:hover {
+    background-color: $color_instaloader_accent;
+    border-color: $color_instaloader_accent;
+  }
+}
@@ -1,28 +0,0 @@
-code {
-    color: #222;
-    background-color: #f5f5f5;
-}
-
-dd {
-    padding-left: 8px;
-}
-
-dl.option dd {
-    padding-left: 12px;
-}
-
-a {
-    color: #008d06;
-}
-
-a:hover {
-    color: #f48400;
-}
-
-body {
-    font-size: 16px;
-}
-
-.versionmodified {
-    font-size: 14px;
-}
@@ -1,14 +0,0 @@
-<li class="dropdown globaltoc-container">
-  <a role="button"
-     id="dLabelGlobalToc"
-     data-toggle="dropdown"
-     data-target="#"
-     href="{{ pathto(master_doc) }}">{{ theme_navbar_site_name }} <b class="caret"></b></a>
-  <ul class="dropdown-menu globaltoc"
-      role="menu"
-      aria-labelledby="dLabelGlobalToc">
-    {{ toctree(maxdepth=theme_globaltoc_depth|toint, collapse=False, includehidden=theme_globaltoc_includehidden|tobool) }}
-    <li role="separator" class="divider"></li>
-    <li class="toctree-l1"><a class="reference internal" href="genindex.html">Index</a></li>
-  </ul>
-</li>
@@ -1,18 +1,115 @@
 {% extends "!layout.html" %}
+
+{% set html5_doctype = True %}
+
+{% set script_files = script_files + ['_static/bootstrap-4.1.3.bundle.min.js', '_static/instaloader.js'] %}
+
 {% block htmltitle %}
  {% if pagename == "index" %}
-  <title>{{ title|striptags|e }}</title>
+    <title>{{ title|striptags|e }}</title>
  {% else %}
-  <title>{{ title|striptags|e }}{{ titlesuffix }}</title>
+    <title>{{ title|striptags|e }}{{ titlesuffix }}</title>
  {% endif %}
 {% endblock %}
+
 {% block extrahead %}
-{{ super() }}
-  <meta name="google-site-verification" content="FNUsDdkOJM9obm9QEdrNBVd9AtAzHzA-aaSMCUrS6C8" />
+  {{ super() }}
+  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
+  <meta name="google-site-verification" content="FNUsDdkOJM9obm9QEdrNBVd9AtAzHzA-aaSMCUrS6C8"/>
  {% if pagename == "index" %}
-  <link rel="canonical" href="https://instaloader.github.io/" />
+    <link rel="canonical" href="https://instaloader.github.io/"/>
  {% else %}
-  <link rel="canonical" href="https://instaloader.github.io/{{ pagename }}.html" />
+    <link rel="canonical" href="https://instaloader.github.io/{{ pagename }}.html"/>
  {% endif %}
-  <script async defer src="https://buttons.github.io/buttons.js"></script>
 {% endblock %}
+
+{% block header %}
+  <nav class="navbar navbar-expand-sm navbar-light bg-light fixed-top border-bottom">
+    <a class="navbar-brand" href="{{ pathto(master_doc) }}">
+      <img src="{{ pathto('_static/' + logo, 1) }}" width="30" height="30" class="d-inline-block align-top" alt="logo">
+      Instaloader
+    </a>
+    <ul class="navbar-nav mr-auto">
+      <li class="nav-item ml-0 ml-sm-2 ml-md-0">
+        <a target="_blank" class="nav-link"
+           href="https://github.com/instaloader/instaloader/releases/tag/v{{ current_release }}"
+           title="Released on {{ current_release_date }}" data-toggle="tooltip">v<b>{{ current_release }}</b></a>
+      </li>
+    </ul>
+    <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarLinks"
+            aria-controls="navbarLinks" aria-expanded="false" aria-label="Toggle navigation">
+      <span class="navbar-toggler-icon"></span>
+    </button>
+    <div class="collapse navbar-collapse" id="navbarLinks">
+      <ul class="navbar-nav ml-auto">
+        <li class="nav-item">
+          <a target="_blank" class="nav-link" href="https://github.com/instaloader/instaloader/issues">Issues</a>
+        </li>
+        <li class="nav-item">
+          <a target="_blank" class="nav-link" href="https://github.com/instaloader/instaloader/releases">Releases</a>
+        </li>
+        <li class="nav-item">
+          <a target="_blank" class="nav-link" href="https://github.com/instaloader/instaloader">GitHub</a>
+        </li>
+      </ul>
+      <form class="form-inline" action="search.html" method="get">
+        <div class="input-group my-2 my-sm-0 ml-2">
+          <input class="form-control" type="search" placeholder="Search" aria-label="Search" name="q">
+          <div class="input-group-append">
+            <button class="btn btn-success btn-small" type="submit">Go</button>
+          </div>
+        </div>
+      </form>
+    </div>
+    {% if pagename != "index" %}
+      <button type="button" class="btn btn-success d-md-none btn-small ml-2" data-toggle="modal"
+              data-target="#navbarToc">
+        TOC
+      </button>
+    {% endif %}
+  </nav>
+  {% if pagename != "index" %}
+    <div class="modal fade" id="navbarToc" tabindex="-1" role="dialog"
+         aria-hidden="true" aria-labelledby="navbarTocTitle">
+      <div class="modal-dialog modal-dialog-centered" role="document">
+        <div class="modal-content">
+          <div class="modal-header">
+            <h5 class="modal-title" id="navbarTocTitle">Contents</h5>
+            <button type="button" class="close" data-dismiss="modal" aria-label="Close">
+              <span aria-hidden="true">&times;</span>
+            </button>
+          </div>
+          <div class="modal-body">
+            {{ toctree(maxdepth=-1) }}
+          </div>
+        </div>
+      </div>
+    </div>
+  {% endif %}
+{% endblock %}
+
+{% block relbar1 %}{% endblock %}
+{% block relbar2 %}{% endblock %}
+{% block footer %}{% endblock %}
+{% block content %}
+  <div class="container-fluid row">
+    <div class="col-12 {% if pagename != "index" %}col-md-9 col-xl-10{% endif %} doc-content">
+      {% block body %}{% endblock %}
+    </div>
+    {% if pagename != "index" %}
+      <div class="d-none col-md-3 col-xl-2 bg-light doc-sidebar d-md-flex flex-column border-left">
+
+        <h3>{{ _('Table of Contents') }}</h3>
+        {{ toctree(maxdepth=-1) }}
+
+        <div class="d-flex justify-content-between small mt-auto mb-2">
+          <a class="reference internal{% if pagename == "genindex" %} current{% endif %}" href="genindex.html">Index</a>
+          {% if pagename != "genindex" and pagename != "search" %}
+            <a class="reference external"
+               href="https://github.com/instaloader/instaloader/edit/master/docs/{{ pagename }}.rst">Edit this page</a>
+          {% endif %}
+        </div>
+      </div>
+    {% endif %}
+  </div>
+{% endblock %}
@@ -1,60 +0,0 @@
-  <div id="navbar" class="{{ theme_navbar_class }} navbar-default {% if theme_navbar_fixed_top|tobool -%} navbar-fixed-top{%- endif -%}">
-    <div class="container">
-      <div class="navbar-header">
-        <!-- .btn-navbar is used as the toggle for collapsed navbar content -->
-        <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse">
-          <span class="icon-bar"></span>
-          <span class="icon-bar"></span>
-          <span class="icon-bar"></span>
-        </button>
-        <a class="navbar-brand" href="{{ pathto(master_doc) }}">
-          {%- block sidebarlogo %}
-            {%- if logo %}<span><img src="{{ pathto('_static/' + logo, 1) }}"></span>{%- endif %}
-          {%- endblock %}
-          {% if theme_navbar_title -%}{{ theme_navbar_title|e }}{%- else -%}{{ project|e }}{%- endif -%}
-        </a>
-        <span class="navbar-text navbar-version pull-left">
-          <a href="https://github.com/instaloader/instaloader/releases/tag/v{{current_release}}" title="Released on {{current_release_date}}">v<b>{{current_release}}</b></a>
-        </span>
-      </div>
-
-        <div class="collapse navbar-collapse nav-collapse">
-          <ul class="nav navbar-nav">
-            {% if theme_navbar_links %}
-              {%- for link in theme_navbar_links %}
-                <li><a href="{{ pathto(*link[1:]) }}">{{ link[0] }}</a></li>
-              {%- endfor %}
-            {% endif %}
-            {% block navbartoc %}
-              {% include "globaltoc.html" %}
-              {% if theme_navbar_pagenav and pagename not in ("index", "genindex", "installation") %}
-                {% include "navbartoc.html" %}
-              {% endif %}
-            {% endblock %}
-            {% if theme_navbar_sidebarrel %}
-              {% block sidebarrel %}
-                {% include "relations.html" %}
-              {% endblock %}
-            {% endif %}
-            </ul>
-            <ul class="nav navbar-nav navbar-right">
-              <li class="dropdown">
-                <a role="button" id="dLabelUsefulLinks" data-toggle="dropdown" data-target="#" href="#">
-                  Useful Links
-                  <b class="caret"></b>
-                </a>
-                <ul class="dropdown-menu" role="menu" aria-labelledby="dLabelUsefulLinks">
-                  <li><a href="https://github.com/instaloader/instaloader/issues">Issue Tracker</a></li>
-                  <li><a href="https://github.com/instaloader/instaloader/releases">Version History</a></li>
-                  <li><a href="https://pypi.org/project/instaloader/">Instaloader on PyPI</a></li>
-                  <li><a href="https://github.com/instaloader/instaloader">Instaloader on GitHub</a></li>
-                  <li role="separator" class="divider"></li>
-                  <li style="padding-left: 20px;"><a class="github-button" href="https://github.com/instaloader/instaloader" data-icon="octicon-star" data-size="large" data-show-count="true" aria-label="Star Instaloader on GitHub">Star</a></li>
-                  <li role="separator" class="divider"></li>
-                  <li><a href="https://www.paypal.me/aandergr"><b>Donate (via Paypal)</b></a></li>
-                </ul>
-              </li>
-            </ul>
-        </div>
-    </div>
-  </div>
@@ -0,0 +1,28 @@
+{% extends "!search.html" %}
+
+{% block body %}
+  <div id="fallback" class="admonition warning">
+    <script type="text/javascript">$('#fallback').hide();</script>
+    <p>
+      {% trans %}Please activate JavaScript to enable the search
+        functionality.{% endtrans %}
+    </p>
+  </div>
+  {% if search_performed %}
+    <h1>{{ _('Search Results') }}</h1>
+    {% if not search_results %}
+      <p>{{ _('Your search did not match any documents. Please make sure that all words are spelled correctly and that you\'ve selected enough categories.') }}</p>
+    {% endif %}
+  {% endif %}
+  <div id="search-results">
+    {% if search_results %}
+      <ul>
+        {% for href, caption, context in search_results %}
+          <li><a href="{{ pathto(item.href) }}">{{ caption }}</a>
+            <div class="context">{{ context|e }}</div>
+          </li>
+        {% endfor %}
+      </ul>
+    {% endif %}
+  </div>
+{% endblock %}
@@ -188,6 +188,10 @@ Exceptions

 .. autoexception:: LoginRequiredException

+.. autoexception:: TwoFactorAuthRequiredException
+
+   .. versionadded:: 4.2
+
 .. autoexception:: InvalidArgumentException

 .. autoexception:: BadResponseException
@@ -85,6 +85,13 @@ Instaloader supports the following targets:
 - ``"#hashtag"``
   Posts with a certain **hashtag** (the quotes are usually necessary),

+- ``%location id``
+   Posts tagged with a given location; the location ID is the numerical ID
+   Instagram labels a location with (e.g.
+   \https://www.instagram.com/explore/locations/**362629379**/plymouth-naval-memorial/).
+
+   .. versionadded:: 4.2
+
 - ``:stories``
   The currently-visible **stories** of your followees (requires
   :option:`--login`),
@@ -126,7 +133,7 @@ Filename Specification
 ^^^^^^^^^^^^^^^^^^^^^^

 For each target, Instaloader creates a directory named after the target,
-i.e. ``profile``, ``#hashtag``, ``:feed``, etc. and therein saves the
+i.e. ``profile``, ``#hashtag``, ``%location id``, ``:feed``, etc. and therein saves the
 posts in files named after the post's timestamp.

 :option:`--dirname-pattern` allows to configure the directory name of each
@@ -192,11 +199,6 @@ where the attributes from :class:`Post` or
 Id est, the following attributes can be used with both
 :option:`--post-filter` and :option:`--storyitem-filter`:

- :attr:`~Post.is_video` (bool)
-   Post/StoryItem is a video. For example, you may skip videos::
-
-      instaloader --post-filter="not is_video" target
-
 - :attr:`~Post.owner_username` (str), :attr:`~Post.owner_id` (int)
   Owner profile username / userid.

@@ -206,6 +208,15 @@ Id est, the following attributes can be used with both

      instaloader --post-filter="date_utc <= datetime(2018, 5, 31)" target

+- :attr:`~Post.is_video` (bool)
+   Post/StoryItem is a video. For example, you may skip videos::
+
+      instaloader --post-filter="not is_video" target
+
+   This is not the same as :option:`--no-videos` and
+   :option:`--no-video-thumbnails`, since sidecar posts (posts that contain
+   multiple pictures/videos in one post) have this attribute set to False.
+
 As :option:`--post-filter`, the following attributes can be used additionally:

 - :attr:`~Post.viewer_has_liked` (bool)
@@ -8,7 +8,7 @@ Instaloader is invoked with::
   $ instaloader [options] target [target ...]

 where ``target`` is a ``profile``, a ``"#hashtag"``, ``@profile`` (all profiles
-that *profile* is following), or if logged in ``:feed`` (pictures from your
+that *profile* is following), ``%location ID``, or if logged in ``:feed`` (pictures from your
 feed), ``:stories`` (stories of your followees) or ``:saved`` (collection of
 posts marked as saved).

@@ -171,7 +171,7 @@ Which Posts to Download
 .. option:: --count COUNT, -c

   Do not attempt to download more than COUNT posts.  Applies only to
-   ``#hashtag`` and ``:feed``.
+   ``#hashtag``, ``%location id``, and ``:feed``.


 Login (Download Private Profiles)
@@ -230,6 +230,14 @@ How to Download
   to ``3``. If a connection fails, it can be manually skipped by hitting
   :kbd:`Control-c`. Set this to ``0`` to retry infinitely.

+.. option:: --commit-mode
+
+   Tries to ensure downloaded images avoid corruption in case of unexpected
+   interruption. If the last picture is corrupted, Instaloader will fix the
+   picture the next time it is run.
+
+   .. versionadded:: 4.2
+
 Miscellaneous Options
 ^^^^^^^^^^^^^^^^^^^^^

@@ -241,18 +249,20 @@ Miscellaneous Options

 .. option:: +args.txt

-   Read arguments from file `args.txt`, a shortcut to provide argument from
-   file rather than command-line. This provide a convient way to hide login
-   info from CLI. and also can use for simplify managment of long arguments.
+   Read arguments from file `args.txt`, a shortcut to provide arguments from
+   file rather than command-line. This provides a convenient way to hide login
+   info from CLI, and can also be used to simplify managment of long arguments.

   .. note::

-      text file should separate arg with line break.
+      Text file should separate arguments with line breaks.

      args.txt example::

-         --login MYUSENAME
-         --password MYPASSWORD
+         --login=MYUSERNAME
+         --password=MYPASSWORD
         --fast-update
+         profile1
+         profile2

   .. versionadded:: 4.1
@@ -14,6 +14,9 @@ Here we present code examples that use the :ref:`python-module-instaloader` for
 more advanced Instagram downloading or metadata mining than what is possible
 with the Instaloader command line interface.

+The scripts presented here can be downloaded from our source tree:
+`instaloader/docs/codesnippets/ <https://github.com/instaloader/instaloader/tree/master/docs/codesnippets>`__
+
 .. For each code snippet:
   - title
   - brief description of what it does / motivation / how it works
@@ -79,6 +82,15 @@ See also :class:`Post`, :meth:`Instaloader.download_post`,

 Discussed in :issue:`113`.

+Top X Posts of User
+-------------------
+
+With Instaloader, it is easy to download the few most-liked pictres of a user.
+
+.. literalinclude:: codesnippets/194_top_x_of_user.py
+
+Discussed in :issue:`194`.
+
 Upgrade Images by Local Copies
 ------------------------------

@@ -0,0 +1,15 @@
+from itertools import islice
+from math import ceil
+
+from instaloader import Instaloader, Profile
+
+PROFILE = ...        # profile to download from
+X_percentage = 10    # percentage of posts that should be downloaded
+
+L = Instaloader()
+
+profile = Profile.from_username(L.context, PROFILE)
+posts_sorted_by_likes = sorted(profile.get_posts(), key = lambda p: p.likes + p.comments)
+
+for post in islice(posts_sorted_by_likes, ceil(profile.mediacount * X_percentage / 100)):
+    L.download_post(post, PROFILE)
@@ -0,0 +1,23 @@
+from glob import glob
+from os.path import expanduser
+from sqlite3 import connect
+
+from instaloader import ConnectionException, Instaloader
+
+# FIREFOXCOOKIEFILE = "/home/alex/.mozilla/firefox/l96w6b90.default/cookies.sqlite"
+FIREFOXCOOKIEFILE = glob(expanduser("~/.mozilla/firefox/*.default/cookies.sqlite"))[0]
+
+instaloader = Instaloader(max_connection_attempts=1)
+instaloader.context._session.cookies.update(connect(FIREFOXCOOKIEFILE)
+                                            .execute("SELECT name, value FROM moz_cookies "
+                                                     "WHERE baseDomain='instagram.com'"))
+
+try:
+    username = instaloader.test_login()
+    if not username:
+        raise ConnectionException()
+except ConnectionException:
+    raise SystemExit("Cookie import failed. Are you logged in successfully in Firefox?")
+
+instaloader.context.username = username
+instaloader.save_session_to_file()
@@ -0,0 +1,4 @@
+# Advanced Instaloader Examples
+
+This directory contains the scripts that are presented at:
+https://instaloader.github.io/codesnippets.html
@@ -20,8 +20,7 @@
 import os
 import subprocess
 import sys
-
-import sphinx_bootstrap_theme
+import typing

 sys.path.insert(0, os.path.abspath('..'))

@@ -123,7 +122,7 @@ add_module_names = False
 # show_authors = False

 # The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
+pygments_style = 'fruity'

 # A list of ignored prefixes for module index sorting.
 # modindex_common_prefix = []
@@ -140,20 +139,7 @@ todo_include_todos = False
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'bootstrap'
-html_theme_path = sphinx_bootstrap_theme.get_html_theme_path()
-
-# Theme options are theme-specific and customize the look and feel of a theme
-# further.  For a list of options available for each theme, see the
-# documentation.
-#
-html_theme_options = {
-    'navbar_site_name': 'Site Contents',
-    'navbar_pagenav_name': 'Page Contents',
-    'navbar_pagenav': True,
-    'navbar_sidebarrel': True,
-    'nosidebar': True,
-     }
+html_theme = 'basic'

 # Add any paths that contain custom themes here, relative to this directory.
 # html_theme_path = []
@@ -374,6 +360,7 @@ html_context = {'current_release': current_release, 'current_release_date': curr


 def setup(app):
-    app.add_stylesheet("style.css")
+    typing.TYPE_CHECKING = True
+    app.add_stylesheet("instaloader.css")
    app.connect('autodoc-process-signature', sphinx_autodoc_typehints.process_signature)
    app.connect('autodoc-process-docstring', sphinx_autodoc_typehints.process_docstring)
@@ -17,12 +17,14 @@ Reporting Bugs
 If you encounter a bug, do not hesitate to report it in our
 `Issue Tracker <https://github.com/instaloader/instaloader/issues>`__. Since
 Instaloader is actively developed, the majority of bugs is fixed within only
-**3 days** after being reported. When reporting a problem, please keep the
+**4 days** after being reported. When reporting a problem, please keep the
 following in mind:

 - Ensure you **use the latest version** of Instaloader. The currently-installed
  version can be found out with ``instaloader --version``.

+- Check whether there is a valid solution in our :ref:`troubleshooting` section.
+
 - Briefly **check whether the bug has already been reported**. If you find an
  issue reporting the same bug you encountered, comment there rather than
  opening a new issue. However, if unsure, please create a new issue.
@@ -43,7 +43,7 @@ See :ref:`install` for more options on how to install Instaloader.

    instaloader [--comments] [--geotags] [--stories] [--highlights] [--tagged]
                [--login YOUR-USERNAME] [--fast-update]
-                profile | "#hashtag" |
+                profile | "#hashtag" | %location_id |
                :stories | :feed | :saved

 See :ref:`download-pictures-from-instagram` for a detailed introduction on how
@@ -60,6 +60,7 @@ Instaloader Documentation
   cli-options
   as-module
   codesnippets
+   troubleshooting
   contributing

 Useful Links
@@ -1,3 +1,2 @@
 requests
 sphinx!=1.7.7
-sphinx-bootstrap-theme
@@ -1,4 +1,4 @@
-# sphinx_autodoc_typehints from Mar 18 2018.
+# sphinx_autodoc_typehints from Nov 20 2018 (version 1.5.2).
 #  https://github.com/agronholm/sphinx-autodoc-typehints
 # (slightly modified to fix class links, maybe related to agronholm/sphinx-autodoc-typehints#38)
 #
@@ -27,8 +27,8 @@
 import inspect
 from typing import get_type_hints, TypeVar, Any, AnyStr, Generic, Union

-from sphinx.util.inspect import getargspec
-from sphinx.ext.autodoc import formatargspec
+from sphinx.util import logging
+from sphinx.util.inspect import Signature

 try:
    from inspect import unwrap
@@ -51,6 +51,8 @@ except ImportError:
            memo.add(id_func)
        return func

+logger = logging.getLogger(__name__)
+

 def format_annotation(annotation):
    if inspect.isclass(annotation) and annotation.__module__ == 'builtins':
@@ -162,23 +164,37 @@ def process_signature(app, what: str, name: str, obj, options, signature, return
        return

    obj = unwrap(obj)
-    try:
-        argspec = getargspec(obj)
-    except (TypeError, ValueError):
-        return
+    signature = Signature(obj)
+    parameters = [
+        param.replace(annotation=inspect.Parameter.empty)
+        for param in signature.signature.parameters.values()
+    ]

-    if argspec.args:
+    if parameters:
        if what in ('class', 'exception'):
-            del argspec.args[0]
+            del parameters[0]
        elif what == 'method':
            outer = inspect.getmodule(obj)
            for clsname in obj.__qualname__.split('.')[:-1]:
                outer = getattr(outer, clsname)
-            method_object = outer.__dict__[obj.__name__]
-            if not isinstance(method_object, (classmethod, staticmethod)):
-                del argspec.args[0]

-    return formatargspec(obj, *argspec[:-1]), None
+            method_name = obj.__name__
+            if method_name.startswith("__") and not method_name.endswith("__"):
+                # If the method starts with double underscore (dunder)
+                # Python applies mangling so we need to prepend the class name.
+                # This doesn't happen if it always ends with double underscore.
+                class_name = obj.__qualname__.split('.')[-2]
+                method_name = "_{c}{m}".format(c=class_name, m=method_name)
+
+            method_object = outer.__dict__[method_name]
+            if not isinstance(method_object, (classmethod, staticmethod)):
+                del parameters[0]
+
+    signature.signature = signature.signature.replace(
+        parameters=parameters,
+        return_annotation=inspect.Signature.empty)
+
+    return signature.format_args().replace('\\', '\\\\'), None


 def process_docstring(app, what, name, obj, options, lines):
@@ -195,8 +211,15 @@ def process_docstring(app, what, name, obj, options, lines):
        except (AttributeError, TypeError):
            # Introspecting a slot wrapper will raise TypeError
            return
+        except NameError as exc:
+            logger.warning('Cannot resolve forward reference in type annotations of "%s": %s',
+                           name, exc)
+            type_hints = obj.__annotations__

        for argname, annotation in type_hints.items():
+            if argname.endswith('_'):
+                argname = '{}\\_'.format(argname[:-1])
+
            formatted_annotation = format_annotation(annotation)

            if argname == 'return':
@@ -0,0 +1,60 @@
+.. _troubleshooting:
+
+Troubleshooting
+===============
+
+.. highlight:: python
+
+429 - Too Many Requests
+-----------------------
+
+Instaloader has a logic to keep track of its requests to Instagram and to obey
+their rate limits. Since they are nowhere documented, we try them out
+experimentally. We have a daily cron job running to confirm that Instaloader
+still stays within the rate limits. Nevertheless, the rate control logic assumes
+that
+
+- at one time, Instaloader is the only application that consumes requests. I.e.
+  neither the Instagram browser interface, nor a mobile app, nor another
+  Instaloader instance is running in parallel,
+
+- no requests had been consumed when Instaloader starts.
+
+The latter one implies that restarting or reinstantiating Instaloader often
+within short time is prone to cause a 429. When a request is denied with a 429,
+Instaloader retries the request as soon as the temporary ban is assumed to be
+expired. In case the retry continuously fails for some reason, which should not
+happen in normal conditions, consider adjusting the
+:option:`--max-connection-attempts` option.
+
+**"Too many queries in the last time"** is not an error. It is a notice that the
+rate limit has almost been reached, according to Instaloader's own rate
+accounting mechanism. We regularly adjust this mechanism to match Instagram's
+current rate limiting.
+
+Login error
+-----------
+
+Instaloader's login *should* work fine, both with and without
+Two-Factor-Authentication. It also supports handling the *checkpoint challenge*,
+issued when Instagram suspects authentication activity on your account, by
+pointing the user to an URL to be opened in a browser.
+
+Nevertheless, in :issue:`92` users report problems with logging in. To still use
+Instaloader's logged-in functionality, you may use the following script to
+workaround login problems by importing the session cookies from Firefox and
+bypassing Instaloader's login.
+
+.. literalinclude:: codesnippets/92_import_firefox_session.py
+
+To use this,
+
+#. login to Instagram in Firefox, 
+
+#. execute the snippet,
+
+#. then, ``instaloader -l USERNAME`` should work fine.
+
+If you do not use your default firefox profile, or your operating system has the
+paths differently set up, you may have to alter the ``FIREFOXCOOKIEFILE``
+variable first.
@@ -1,7 +1,7 @@
 """Download pictures (or videos) along with their captions and other metadata from Instagram."""


-__version__ = '4.1b1'
+__version__ = '4.2.1'


 try:
@@ -8,7 +8,8 @@ from argparse import ArgumentParser, SUPPRESS
 from typing import List, Optional

 from . import (Instaloader, InstaloaderException, InvalidArgumentException, Post, Profile, ProfileNotExistsException,
-               StoryItem, __version__, load_structure_from_file)
+               StoryItem, __version__, load_structure_from_file, TwoFactorAuthRequiredException,
+               BadCredentialsException)
 from .instaloader import get_default_session_filename
 from .instaloadercontext import default_user_agent

@@ -20,7 +21,7 @@ def usage_string():
    return """
 {0} [--comments] [--geotags] [--stories] [--highlights] [--tagged]
 {2:{1}} [--login YOUR-USERNAME] [--fast-update]
-{2:{1}} profile | "#hashtag" | :stories | :feed | :saved
+{2:{1}} profile | "#hashtag" | %%location_id | :stories | :feed | :saved
 {0} --help""".format(argv0, len(argv0), '')


@@ -84,7 +85,16 @@ def _main(instaloader: Instaloader, targetlist: List[str],
            instaloader.context.log("Session file does not exist yet - Logging in.")
        if not instaloader.context.is_logged_in or username != instaloader.test_login():
            if password is not None:
-                instaloader.login(username, password)
+                try:
+                    instaloader.login(username, password)
+                except TwoFactorAuthRequiredException:
+                    while True:
+                        try:
+                            code = input("Enter 2FA verification code: ")
+                            instaloader.two_factor_login(code)
+                            break
+                        except BadCredentialsException:
+                            pass
            else:
                instaloader.interactive_login(username)
        instaloader.context.log("Logged in as %s." % username)
@@ -130,6 +140,9 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                                                 post_filter=post_filter)
                elif target[0] == '-':
                    instaloader.download_post(Post.from_shortcode(instaloader.context, target[1:]), target)
+                elif target[0] == "%":
+                    instaloader.download_location(location=target[1:], max_count=max_count, fast_update=fast_update,
+                                                  post_filter=post_filter)
                elif target == ":feed":
                    instaloader.download_feed_posts(fast_update=fast_update, max_count=max_count,
                                                    post_filter=post_filter)
@@ -166,6 +179,8 @@ def _main(instaloader: Instaloader, targetlist: List[str],
        if len(profiles) > 1:
            instaloader.context.log("Downloading {} profiles: {}".format(len(profiles),
                                                                         ' '.join([p.username for p in profiles])))
+        if profiles and download_profile_pic and not instaloader.context.is_logged_in:
+            instaloader.context.error("Warning: Use --login to download HD version of profile pictures.")
        instaloader.download_profiles(profiles,
                                      download_profile_pic, download_posts, download_tagged, download_highlights,
                                      download_stories, fast_update, post_filter, storyitem_filter)
@@ -208,6 +223,7 @@ def main():
                           help="Download all followees of profile. Requires --login. "
                                "Consider using :feed rather than @yourself.")
    g_targets.add_argument('_hashtag', nargs='*', metavar='"#hashtag"', help="Download #hashtag.")
+    g_targets.add_argument('_location', nargs='*', metavar='%location_id', help="Download %%location_id.")
    g_targets.add_argument('_feed', nargs='*', metavar=":feed",
                           help="Download pictures from your feed. Requires --login.")
    g_targets.add_argument('_stories', nargs='*', metavar=":stories",
@@ -320,6 +336,10 @@ def main():
                       help='Maximum number of connection attempts until a request is aborted. Defaults to 3. If a '
                            'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry '
                            'infinitely.')
+    g_how.add_argument('--commit-mode', action='store_true',
+                       help='Tries to ensure downloaded images avoid corruption in case of unexpected interruption. '
+                       'If the last picture is corrupted, Instaloader will fix the picture the next time it is run. '
+                       'Requires the JSON metadata to be saved.')

    g_misc = parser.add_argument_group('Miscellaneous Options')
    g_misc.add_argument('-q', '--quiet', action='store_true',
@@ -361,6 +381,9 @@ def main():
        download_posts = not (args.no_posts or args.stories_only or args.profile_pic_only)
        download_stories = args.stories or args.stories_only

+        if args.commit_mode and args.no_metadata_json:
+            raise SystemExit('--commit-mode requires JSON metadata to be saved.')
+
        loader = Instaloader(sleep=not args.no_sleep, quiet=args.quiet, user_agent=args.user_agent,
                             dirname_pattern=args.dirname_pattern, filename_pattern=args.filename_pattern,
                             download_pictures=not args.no_pictures,
@@ -371,7 +394,8 @@ def main():
                             post_metadata_txt_pattern=post_metadata_txt_pattern,
                             storyitem_metadata_txt_pattern=storyitem_metadata_txt_pattern,
                             graphql_rate_limit=args.graphql_rate_limit,
-                             max_connection_attempts=args.max_connection_attempts)
+                             max_connection_attempts=args.max_connection_attempts,
+                             commit_mode=args.commit_mode)
        _main(loader,
              args.profile,
              username=args.login.lower() if args.login is not None else None,
@@ -33,6 +33,10 @@ class LoginRequiredException(InstaloaderException):
    pass


+class TwoFactorAuthRequiredException(InstaloaderException):
+    pass
+
+
 class InvalidArgumentException(InstaloaderException):
    pass

@@ -1,5 +1,6 @@
 import getpass
 import json
+import lzma
 import os
 import platform
 import re
@@ -10,12 +11,13 @@ import tempfile
 from contextlib import contextmanager, suppress
 from datetime import datetime, timezone
 from functools import wraps
+from hashlib import md5
 from io import BytesIO
 from typing import Any, Callable, Iterator, List, Optional, Set, Union

 from .exceptions import *
 from .instaloadercontext import InstaloaderContext
-from .structures import Highlight, JsonExportable, Post, PostLocation, Profile, Story, StoryItem, save_structure_to_file
+from .structures import Highlight, JsonExportable, Post, PostLocation, Profile, Story, StoryItem, save_structure_to_file, load_structure_from_file


 def get_default_session_filename(username: str) -> str:
@@ -95,6 +97,7 @@ class Instaloader:
       txt file.
    :param storyitem_metadata_txt_pattern: :option:`--storyitem-metadata-txt`, default is empty (=none)
    :param max_connection_attempts: :option:`--max-connection-attempts`
+    :param commit_mode: :option:`--commit-mode`

    .. attribute:: context

@@ -116,7 +119,8 @@ class Instaloader:
                 post_metadata_txt_pattern: str = None,
                 storyitem_metadata_txt_pattern: str = None,
                 graphql_rate_limit: Optional[int] = None,
-                 max_connection_attempts: int = 3):
+                 max_connection_attempts: int = 3,
+                 commit_mode: bool = False):

        self.context = InstaloaderContext(sleep, quiet, user_agent, graphql_rate_limit, max_connection_attempts)

@@ -134,6 +138,12 @@ class Instaloader:
            else post_metadata_txt_pattern
        self.storyitem_metadata_txt_pattern = '' if storyitem_metadata_txt_pattern is None \
            else storyitem_metadata_txt_pattern
+        self.commit_mode = commit_mode
+        if self.commit_mode and not self.save_metadata:
+            raise InvalidArgumentException("Commit mode requires JSON metadata to be saved.")
+
+        # Used to keep state in commit mode
+        self._committed = None

    @contextmanager
    def anonymous_copy(self):
@@ -173,9 +183,15 @@ class Instaloader:
        if filename_suffix is not None:
            filename += '_' + filename_suffix
        filename += '.' + file_extension
-        if os.path.isfile(filename):
-            self.context.log(filename + ' exists', end=' ', flush=True)
-            return False
+        # A post is considered "commited" if the json file exists and is not malformed.
+        if self.commit_mode:
+            if self._committed and os.path.isfile(filename):
+                self.context.log(filename + ' exists', end=' ', flush=True)
+                return False
+        else:
+            if os.path.isfile(filename):
+                self.context.log(filename + ' exists', end=' ', flush=True)
+                return False
        self.context.get_and_write_raw(url, filename)
        os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))
        return True
@@ -239,8 +255,7 @@ class Instaloader:
                return None
            else:
                def get_filename(index):
-                    return filename if index == 0 else (filename[:-4] + '_old_' +
-                                                        (str(0) if index < 10 else str()) + str(index) + filename[-4:])
+                    return filename if index == 0 else '{0}_old_{2:02}{1}'.format(*os.path.splitext(filename), index)

                i = 0
                while os.path.isfile(get_filename(i)):
@@ -274,26 +289,34 @@ class Instaloader:
        """Downloads and saves profile pic."""

        def _epoch_to_string(epoch: datetime) -> str:
-            return epoch.strftime('%Y-%m-%d_%H-%M-%S')
+            return epoch.strftime('%Y-%m-%d_%H-%M-%S_UTC')

-        profile_pic_url = profile.profile_pic_url
-        with self.context.get_anonymous_session() as anonymous_session:
-            date_object = datetime.strptime(anonymous_session.head(profile_pic_url).headers["Last-Modified"],
-                                            '%a, %d %b %Y %H:%M:%S GMT')
+        profile_pic_response = self.context.get_raw(profile.profile_pic_url)
+        if 'Last-Modified' in profile_pic_response.headers:
+            date_object = datetime.strptime(profile_pic_response.headers["Last-Modified"], '%a, %d %b %Y %H:%M:%S GMT')
+            profile_pic_bytes = None
+            profile_pic_identifier = _epoch_to_string(date_object)
+        else:
+            date_object = None
+            profile_pic_bytes = profile_pic_response.content
+            profile_pic_identifier = md5(profile_pic_bytes).hexdigest()[:16]
        profile_pic_extension = 'jpg'
        if ((format_string_contains_key(self.dirname_pattern, 'profile') or
             format_string_contains_key(self.dirname_pattern, 'target'))):
-            filename = '{0}/{1}_UTC_profile_pic.{2}'.format(self.dirname_pattern.format(profile=profile.username.lower(),
-                                                                                        target=profile.username.lower()),
-                                                            _epoch_to_string(date_object), profile_pic_extension)
+            filename = '{0}/{1}_profile_pic.{2}'.format(self.dirname_pattern.format(profile=profile.username.lower(),
+                                                                                    target=profile.username.lower()),
+                                                        profile_pic_identifier, profile_pic_extension)
        else:
-            filename = '{0}/{1}_{2}_UTC_profile_pic.{3}'.format(self.dirname_pattern.format(), profile.username.lower(),
-                                                                _epoch_to_string(date_object), profile_pic_extension)
-        if os.path.isfile(filename):
+            filename = '{0}/{1}_{2}_profile_pic.{3}'.format(self.dirname_pattern.format(), profile.username.lower(),
+                                                            profile_pic_identifier, profile_pic_extension)
+        content_length = profile_pic_response.headers.get('Content-Length', None)
+        if os.path.isfile(filename) and (not self.context.is_logged_in or
+                                         content_length is not None and os.path.getsize(filename) >= int(content_length)):
            self.context.log(filename + ' already exists')
            return None
-        self.context.get_and_write_raw(profile_pic_url, filename)
-        os.utime(filename, (datetime.now().timestamp(), date_object.timestamp()))
+        self.context.write_raw(profile_pic_bytes if profile_pic_bytes else profile_pic_response, filename)
+        if date_object:
+            os.utime(filename, (datetime.now().timestamp(), date_object.timestamp()))
        self.context.log('')  # log output of _get_and_write_raw() does not produce \n

    @_requires_login
@@ -335,9 +358,20 @@ class Instaloader:

        :raises InvalidArgumentException: If the provided username does not exist.
        :raises BadCredentialsException: If the provided password is wrong.
-        :raises ConnectionException: If connection to Instagram failed."""
+        :raises ConnectionException: If connection to Instagram failed.
+        :raises TwoFactorAuthRequiredException: First step of 2FA login done, now call :meth:`Instaloader.two_factor_login`."""
        self.context.login(user, passwd)

+    def two_factor_login(self, two_factor_code) -> None:
+        """Second step of login if 2FA is enabled.
+        Not meant to be used directly, use :meth:`Instaloader.two_factor_login`.
+
+        :raises InvalidArgumentException: No two-factor authentication pending.
+        :raises BadCredentialsException: 2FA verification code invalid.
+
+        .. versionadded:: 4.2"""
+        self.context.two_factor_login(two_factor_code)
+
    def format_filename(self, item: Union[Post, StoryItem], target: Optional[str] = None):
        """Format filename of a :class:`Post` or :class:`StoryItem` according to ``filename-pattern`` parameter.

@@ -358,18 +392,19 @@ class Instaloader:
        os.makedirs(os.path.dirname(filename), exist_ok=True)

        # Download the image(s) / video thumbnail and videos within sidecars if desired
-        downloaded = False
+        downloaded = True
+        self._committed = self.check_if_committed(filename)
        if self.download_pictures:
            if post.typename == 'GraphSidecar':
                edge_number = 1
                for sidecar_node in post.get_sidecar_nodes():
                    # Download picture or video thumbnail
                    if not sidecar_node.is_video or self.download_video_thumbnails is True:
-                        downloaded |= self.download_pic(filename=filename, url=sidecar_node.display_url,
+                        downloaded &= self.download_pic(filename=filename, url=sidecar_node.display_url,
                                                        mtime=post.date_local, filename_suffix=str(edge_number))
                    # Additionally download video if available and desired
                    if sidecar_node.is_video and self.download_videos is True:
-                        downloaded |= self.download_pic(filename=filename, url=sidecar_node.video_url,
+                        downloaded &= self.download_pic(filename=filename, url=sidecar_node.video_url,
                                                        mtime=post.date_local, filename_suffix=str(edge_number))
                    edge_number += 1
            elif post.typename == 'GraphImage':
@@ -387,7 +422,7 @@ class Instaloader:

        # Download video if desired
        if post.is_video and self.download_videos is True:
-            downloaded |= self.download_pic(filename=filename, url=post.video_url, mtime=post.date_local)
+            downloaded &= self.download_pic(filename=filename, url=post.video_url, mtime=post.date_local)

        # Download geotags if desired
        if self.download_geotags and post.location:
@@ -632,6 +667,59 @@ class Instaloader:
                if fast_update and not downloaded:
                    break

+    def get_location_posts(self, location: str) -> Iterator[Post]:
+        """Get Posts which are listed by Instagram for a given Location.
+
+        :return:  Iterator over Posts of a location's posts
+
+        .. versionadded:: 4.2
+        """
+        has_next_page = True
+        end_cursor = None
+        while has_next_page:
+            if end_cursor:
+                params = {'__a': 1, 'max_id': end_cursor}
+            else:
+                params = {'__a': 1}
+            location_data = self.context.get_json('explore/locations/{0}/'.format(location),
+                                                  params)['graphql']['location']['edge_location_to_media']
+            yield from (Post(self.context, edge['node']) for edge in location_data['edges'])
+            has_next_page = location_data['page_info']['has_next_page']
+            end_cursor = location_data['page_info']['end_cursor']
+
+    def download_location(self, location: str,
+                          max_count: Optional[int] = None,
+                          post_filter: Optional[Callable[[Post], bool]] = None,
+                          fast_update: bool = False) -> None:
+        """Download pictures of one location.
+
+        To download the last 30 pictures with location 362629379, do::
+
+            loader = Instaloader()
+            loader.download_location(362629379, max_count=30)
+
+        :param location: Location to download, as Instagram numerical ID
+        :param max_count: Maximum count of pictures to download
+        :param post_filter: function(post), which returns True if given picture should be downloaded
+        :param fast_update: If true, abort when first already-downloaded picture is encountered
+
+        .. versionadded:: 4.2
+        """
+        self.context.log("Retrieving pictures for location {}...".format(location))
+        count = 1
+        for post in self.get_location_posts(location):
+            if max_count is not None and count > max_count:
+                break
+            self.context.log('[{0:3d}] %{1} '.format(count, location), end='', flush=True)
+            if post_filter is not None and not post_filter(post):
+                self.context.log('<skipped>')
+                continue
+            count += 1
+            with self.context.error_catcher('Download location {}'.format(location)):
+                downloaded = self.download_post(post, target='%' + location)
+                if fast_update and not downloaded:
+                    break
+
    @_requires_login
    def get_explore_posts(self) -> Iterator[Post]:
        """Get Posts which are worthy of exploring suggested by Instagram.
@@ -823,7 +911,7 @@ class Instaloader:
                        self.download_profilepic(profile)

                # Catch some errors
-                if profile.is_private:
+                if profile.is_private and (tagged or highlights or posts):
                    if not self.context.is_logged_in:
                        raise LoginRequiredException("--login=USERNAME required.")
                    if not profile.followed_by_viewer and self.context.username != profile.username:
@@ -945,6 +1033,25 @@ class Instaloader:
                if fast_update and not downloaded:
                    break

+    def check_if_committed(self, filename: str) -> bool:
+        """Checks to see if the current post has been committed.
+
+        A post is considered committed if its json metadata file exists and is not malformed.
+
+        .. versionadded:: 4.2
+        """
+        if os.path.isfile(filename + '.json.xz'):
+            filename += '.json.xz'
+        elif os.path.isfile(filename + '.json'):
+            filename += '.json'
+        else:
+            return False
+        try:
+            load_structure_from_file(self.context, filename)
+            return True
+        except (FileNotFoundError, lzma.LZMAError, json.decoder.JSONDecodeError):
+            return False
+
    def interactive_login(self, username: str) -> None:
        """Logs in and internally stores session, asking user for password interactively.

@@ -953,11 +1060,20 @@ class Instaloader:
        :raises ConnectionException: If connection to Instagram failed."""
        if self.context.quiet:
            raise LoginRequiredException("Quiet mode requires given password or valid session file.")
-        password = None
-        while password is None:
-            password = getpass.getpass(prompt="Enter Instagram password for %s: " % username)
-            try:
-                self.login(username, password)
-            except BadCredentialsException as err:
-                print(err, file=sys.stderr)
-                password = None
+        try:
+            password = None
+            while password is None:
+                password = getpass.getpass(prompt="Enter Instagram password for %s: " % username)
+                try:
+                    self.login(username, password)
+                except BadCredentialsException as err:
+                    print(err, file=sys.stderr)
+                    password = None
+        except TwoFactorAuthRequiredException:
+            while True:
+                try:
+                    code = input("Enter 2FA verification code: ")
+                    self.two_factor_login(code)
+                    break
+                except BadCredentialsException:
+                    pass
@@ -10,7 +10,7 @@ import time
 import urllib.parse
 from contextlib import contextmanager
 from datetime import datetime, timedelta
-from typing import Any, Callable, Dict, Iterator, Optional
+from typing import Any, Callable, Dict, Iterator, Optional, Union

 import requests
 import requests.utils
@@ -57,8 +57,9 @@ class InstaloaderContext:
        self.quiet = quiet
        self.max_connection_attempts = max_connection_attempts
        self._graphql_page_length = 50
-        self.graphql_count_per_slidingwindow = graphql_count_per_slidingwindow or 20
+        self.graphql_count_per_slidingwindow = graphql_count_per_slidingwindow or 200
        self._root_rhx_gis = None
+        self.two_factor_auth_pending = None

        # error log, filled with error() and printed at the end of Instaloader.main()
        self.error_log = []
@@ -180,23 +181,39 @@ class InstaloaderContext:

        :raises InvalidArgumentException: If the provided username does not exist.
        :raises BadCredentialsException: If the provided password is wrong.
-        :raises ConnectionException: If connection to Instagram failed."""
+        :raises ConnectionException: If connection to Instagram failed.
+        :raises TwoFactorAuthRequiredException: First step of 2FA login done, now call :meth:`Instaloader.two_factor_login`."""
        import http.client
        # pylint:disable=protected-access
        http.client._MAXHEADERS = 200
        session = requests.Session()
        session.cookies.update({'sessionid': '', 'mid': '', 'ig_pr': '1',
-                                'ig_vw': '1920', 'csrftoken': '',
+                                'ig_vw': '1920', 'ig_cb': '1', 'csrftoken': '',
                                's_network': '', 'ds_user_id': ''})
        session.headers.update(self._default_http_header())
-        session.headers.update({'X-CSRFToken': self.get_json('', {})['config']['csrf_token']})
+        session.get('https://www.instagram.com/web/__mid/')
+        csrf_token = session.cookies.get_dict()['csrftoken']
+        session.headers.update({'X-CSRFToken': csrf_token})
        # Not using self.get_json() here, because we need to access csrftoken cookie
        self._sleep()
        login = session.post('https://www.instagram.com/accounts/login/ajax/',
                             data={'password': passwd, 'username': user}, allow_redirects=True)
-        if login.status_code != 200:
-            raise ConnectionException("Login error: {} {}".format(login.status_code, login.reason))
-        resp_json = login.json()
+        try:
+            resp_json = login.json()
+        except json.decoder.JSONDecodeError:
+            raise ConnectionException("Login error: JSON decode fail, {} - {}.".format(login.status_code, login.reason))
+        if resp_json.get('two_factor_required'):
+            two_factor_session = copy_session(session)
+            two_factor_session.headers.update({'X-CSRFToken': csrf_token})
+            two_factor_session.cookies.update({'csrftoken': csrf_token})
+            self.two_factor_auth_pending = (two_factor_session,
+                                            user,
+                                            resp_json['two_factor_info']['two_factor_identifier'])
+            raise TwoFactorAuthRequiredException("Login error: two-factor authentication required.")
+        if resp_json.get('checkpoint_url'):
+            raise ConnectionException("Login: Checkpoint required. Point your browser to "
+                                      "https://www.instagram.com{}, "
+                                      "follow the instructions, then retry.".format(resp_json.get('checkpoint_url')))
        if resp_json['status'] != 'ok':
            if 'message' in resp_json:
                raise ConnectionException("Login error: \"{}\" status, message \"{}\".".format(resp_json['status'],
@@ -218,10 +235,36 @@ class InstaloaderContext:
        self._session = session
        self.username = user

+    def two_factor_login(self, two_factor_code):
+        """Second step of login if 2FA is enabled.
+        Not meant to be used directly, use :meth:`Instaloader.two_factor_login`.
+
+        :raises InvalidArgumentException: No two-factor authentication pending.
+        :raises BadCredentialsException: 2FA verification code invalid.
+
+        .. versionadded:: 4.2"""
+        if not self.two_factor_auth_pending:
+            raise InvalidArgumentException("No two-factor authentication pending.")
+        (session, user, two_factor_id) = self.two_factor_auth_pending
+
+        login = session.post('https://www.instagram.com/accounts/login/ajax/two_factor/',
+                             data={'username': user, 'verificationCode': two_factor_code, 'identifier': two_factor_id},
+                             allow_redirects=True)
+        resp_json = login.json()
+        if resp_json['status'] != 'ok':
+            if 'message' in resp_json:
+                raise BadCredentialsException("Login error: {}".format(resp_json['message']))
+            else:
+                raise BadCredentialsException("Login error: \"{}\" status.".format(resp_json['status']))
+        session.headers.update({'X-CSRFToken': login.cookies['csrftoken']})
+        self._session = session
+        self.username = user
+        self.two_factor_auth_pending = None
+
    def _sleep(self):
        """Sleep a short time if self.sleep is set. Called before each request to instagram.com."""
        if self.sleep:
-            time.sleep(min(random.expovariate(0.6), 5.0))
+            time.sleep(min(random.expovariate(0.7), 5.0))

    def get_json(self, path: str, params: Dict[str, Any], host: str = 'www.instagram.com',
                 session: Optional[requests.Session] = None, _attempt=1) -> Dict[str, Any]:
@@ -248,8 +291,7 @@ class InstaloaderContext:
        is_graphql_query = 'query_hash' in params and 'graphql/query' in path
        # some queries are not rate limited if invoked anonymously:
        query_not_limited = is_graphql_query and not self.is_logged_in \
-                            and params['query_hash'] in ['9ca88e465c3f866a76f7adee3871bdd8',
-                                                         '472f257a40c653c64c666ce877d59d2b']
+                            and params['query_hash'] in ['9ca88e465c3f866a76f7adee3871bdd8']
        if is_graphql_query and not query_not_limited:
            waittime = graphql_query_waittime()
            if waittime > 0:
@@ -307,6 +349,7 @@ class InstaloaderContext:
                if isinstance(err, TooManyRequestsException):
                    print(textwrap.fill(text_for_429), file=sys.stderr)
                    if is_graphql_query:
+                        print("Made {} GraphQL requests.".format(len(self.query_timestamps)), file=sys.stderr)
                        waittime = graphql_query_waittime(untracked_queries=True)
                        if waittime > 0:
                            self.log('The request will be retried in {} seconds, at {:%H:%M}.'
@@ -386,8 +429,35 @@ class InstaloaderContext:
            data = _query()
            yield from (edge['node'] for edge in data['edges'])

-    def get_and_write_raw(self, url: str, filename: str, _attempt=1) -> None:
-        """Downloads raw data.
+    def get_iphone_json(self, path: str, params: Dict[str, Any]) -> Dict[str, Any]:
+        """JSON request to ``i.instagram.com``.
+
+        :param path: URL, relative to ``i.instagram.com/``
+        :param params: GET parameters
+        :return: Decoded response dictionary
+        :raises QueryReturnedBadRequestException: When the server responds with a 400.
+        :raises QueryReturnedNotFoundException: When the server responds with a 404.
+        :raises ConnectionException: When query repeatedly failed.
+
+        .. versionadded:: 4.2.1"""
+        tempsession = copy_session(self._session)
+        tempsession.headers['User-Agent'] = 'Instagram 10.3.2 (iPhone7,2; iPhone OS 9_3_3; en_US; en-US; ' \
+                                            'scale=2.00; 750x1334) AppleWebKit/420+'
+        for header in ['Host', 'Origin', 'X-Instagram-AJAX', 'X-Requested-With']:
+            tempsession.headers.pop(header, None)
+        return self.get_json(path, params, 'i.instagram.com', tempsession)
+
+    def write_raw(self, resp: Union[bytes, requests.Response], filename: str) -> None:
+        """Write raw response data into a file."""
+        self.log(filename, end=' ', flush=True)
+        with open(filename, 'wb') as file:
+            if isinstance(resp, requests.Response):
+                shutil.copyfileobj(resp.raw, file)
+            else:
+                file.write(resp)
+
+    def get_raw(self, url: str, _attempt=1) -> requests.Response:
+        """Downloads a file anonymously.

        :raises QueryReturnedNotFoundException: When the server responds with a 404.
        :raises QueryReturnedForbiddenException: When the server responds with a 403.
@@ -396,10 +466,8 @@ class InstaloaderContext:
            with self.get_anonymous_session() as anonymous_session:
                resp = anonymous_session.get(url, stream=True)
            if resp.status_code == 200:
-                self.log(filename, end=' ', flush=True)
-                with open(filename, 'wb') as file:
-                    resp.raw.decode_content = True
-                    shutil.copyfileobj(resp.raw, file)
+                resp.raw.decode_content = True
+                return resp
            else:
                if resp.status_code == 403:
                    # suspected invalid URL signature
@@ -415,11 +483,19 @@ class InstaloaderContext:
            self.error(error_string + " [retrying; skip with ^C]", repeat_at_end=False)
            try:
                self._sleep()
-                self.get_and_write_raw(url, filename, _attempt + 1)
+                return self.get_raw(url, _attempt + 1)
            except KeyboardInterrupt:
                self.error("[skipped by user]", repeat_at_end=False)
                raise ConnectionException(error_string) from err

+    def get_and_write_raw(self, url: str, filename: str) -> None:
+        """Downloads and writes anonymously-requested raw data into a file.
+
+        :raises QueryReturnedNotFoundException: When the server responds with a 404.
+        :raises QueryReturnedForbiddenException: When the server responds with a 403.
+        :raises ConnectionException: When download repeatedly failed."""
+        self.write_raw(self.get_raw(url), filename)
+
    @property
    def root_rhx_gis(self) -> Optional[str]:
        """rhx_gis string returned in the / query."""
@@ -455,10 +455,10 @@ class Profile:

    @property
    def _iphone_struct(self) -> Dict[str, Any]:
+        if not self._context.is_logged_in:
+            raise LoginRequiredException("--login required to access iPhone profile info endpoint.")
        if not self._iphone_struct_:
-            with self._context.anonymous_copy() as anonymous_context:
-                data = anonymous_context.get_json(path='api/v1/users/{}/info/'.format(self.userid),
-                                                  params={}, host='i.instagram.com')
+            data = self._context.get_iphone_json(path='api/v1/users/{}/info/'.format(self.userid), params={})
            self._iphone_struct_ = data['user']
        return self._iphone_struct_

@@ -578,13 +578,19 @@ class Profile:

    @property
    def profile_pic_url(self) -> str:
-        """Return URL of profile picture
+        """Return URL of profile picture. If logged in, the HD version is returned, otherwise a lower-quality version.

-        .. versionadded:: 4.0.3"""
-        try:
-            return self._iphone_struct['hd_profile_pic_url_info']['url']
-        except (InstaloaderException, KeyError) as err:
-            self._context.error('{} Unable to fetch high quality profile pic.'.format(err))
+        .. versionadded:: 4.0.3
+
+        .. versionchanged:: 4.2.1
+           Require being logged in for HD version (as required by Instagram)."""
+        if self._context.is_logged_in:
+            try:
+                return self._iphone_struct['hd_profile_pic_url_info']['url']
+            except (InstaloaderException, KeyError) as err:
+                self._context.error('{} Unable to fetch high quality profile pic.'.format(err))
+                return self._metadata("profile_pic_url_hd")
+        else:
            return self._metadata("profile_pic_url_hd")

    def get_profile_pic_url(self) -> str:
@@ -12,6 +12,7 @@ PROFILE_WITH_HIGHLIGHTS = 325732271
 PUBLIC_PROFILE = "selenagomez"
 PUBLIC_PROFILE_ID = 460563723
 HASHTAG = "kitten"
+LOCATION = "362629379"
 OWN_USERNAME = "aandergr"
 NORMAL_MAX_COUNT = 2
 PAGING_MAX_COUNT = 15
@@ -63,6 +64,21 @@ class TestInstaloaderAnonymously(unittest.TestCase):
    def test_hashtag_download(self):
        self.L.download_hashtag(HASHTAG, NORMAL_MAX_COUNT)

+    def test_hashtag_paging(self):
+        for count, post in enumerate(self.L.get_hashtag_posts(HASHTAG)):
+            print(post)
+            if count == PAGING_MAX_COUNT:
+                break
+
+    def test_location_download(self):
+        self.L.download_location(LOCATION, NORMAL_MAX_COUNT)
+
+    def test_location_paging(self):
+        for count, post in enumerate(self.L.get_location_posts(LOCATION)):
+            print(post)
+            if count == PAGING_MAX_COUNT:
+                break
+
    def test_get_id_by_username(self):
        self.assertEqual(PUBLIC_PROFILE_ID,
                         instaloader.Profile.from_username(self.L.context, PUBLIC_PROFILE).userid)
Author	SHA1	Message	Date
Alexander Graf	57892f191a	Release of version 4.2.1	2019-01-17 17:47:45 +01:00
Alexander Graf	26e74bad6a	Download HD profile picture if logged in Unfortunately, it is now required to be logged in to access the HD version of profile pictures. When attempting to download profile pictures without --login, a warning message is printed once, and the lower-quality versions are obtained. For backwards compatibility, already-downloaded profile pictures are overwritten if the now-obtainable version is assumed to be of better quality than the existing one (determined by file size vs Content-Length). The iPhone endpoint is accessed with code exhumed from `c355338010`. Also, this reverts "Profile: don't access removed iphone info endpoint" `08327c4117`. This fixes #209.	2019-01-17 17:39:29 +01:00
Alexander Graf	27a239f07f	doc theme: Improve consistency of code highlight	2019-01-17 11:39:18 +01:00
Alexander Graf	bb25f03c7d	doc: "Troubleshooting" section + minor changes	2019-01-17 11:00:04 +01:00
Lars Lindqvist	c54342bb98	Refactor `get_filename` Use string formatting instead of `str(0) + str(index)` and don't assume file extension is three characters.	2019-01-08 19:38:46 +01:00
Alexander Graf	d2bebcd00c	doc: Fix +args.txt description - Fix wrong `--option value`, which must be `--option=value` - Fix typos Fixes #221.	2019-01-03 17:05:43 +01:00
Alexander Graf	4bc0a94e12	Name profilepic by hash if Last-Modified missing This fixes #188. Also, this commit prevents double-requesting the profile pic URL to obtain the Last-Modified header.	2019-01-03 16:49:21 +01:00
Alexander Graf	06845b53fc	Fixes and enhancements to documentation theme	2019-01-01 14:23:08 +01:00
Alexander Graf	e9cec0b261	docs/conf.py: remove unneeded import	2018-12-31 19:35:55 +01:00
Alexander Graf	ac4cd9f595	New doc design: Sphinx Bootstrap 4 Theme Now we use an own, mobile-first, responsive sphinx theme based on Bootstrap 4. It makes navigation on the very long "Python module instaloader" page a lot easier. Also, it looks better. This solves all the problems we had with sphinx-bootstrap-theme.	2018-12-31 19:30:19 +01:00
Alexander Graf	2e7835850a	Version 4.2	2018-12-23 17:07:03 +01:00
Alexander Graf	89b56bdfa8	Login: Handle suspicious login and JSON error Motivated by #92.	2018-12-21 22:39:07 +01:00
Alexander Graf	1387d396dd	Second Release Candidate for Version 4.2	2018-12-21 22:01:28 +01:00
André Koch-Kramer	4ee867c61b	Implement two-factor authentication Closes #200.	2018-12-21 21:58:12 +01:00
Alexander Graf	edba6959d9	First Release Candidate for Version 4.2	2018-12-21 17:54:02 +01:00
Alexander Graf	08327c4117	Profile: don't access removed iphone info endpoint Don't access the endpoint that was used to access the HD version of the profile picture. This addresses #209.	2018-12-21 17:06:32 +01:00
Alexander Graf	4ab0fea82a	login: print that 2FA is not supported if required 2FA support is requested in issue #200, and also mentioned in #92.	2018-12-21 16:54:56 +01:00
Alexander Graf	a735b02bac	doc: Add 'versionadded' directives to new features	2018-12-21 14:43:33 +01:00
Stijn Peeters	be5d02ef3b	Location search (#212 ) * Add %location search option Search for posts for a given location ID using %[location id] as the query * Document %location search * Make pylint happy * Use correct paths for location results * Fix —help output Add description of location argument and fix output error for short help. * Add unit tests for location download * Add extra unit test for locations	2018-12-17 21:35:31 +01:00
Alexander Graf	1ab9e44104	Update sphinx_autodoc_typehints to version 1.5.2	2018-12-13 16:23:14 +01:00
André Koch-Kramer	6212d22e26	Merge branch 'master' of https://github.com/sushilicious/instaloader into v4.2-dev	2018-12-13 15:53:06 +01:00
André Koch-Kramer	aa859b9f54	Ignore pylint's cyclic-import and unnecessary-pass	2018-12-13 15:29:46 +01:00
sushilicious	ffda2265bd	Added _committed to __init__ to make pylint shut up	2018-12-09 02:12:45 -05:00
sushilicious	e753406a6c	Just in case, check whether picture exists for commit This is in addition to the other commit logic.	2018-12-09 02:12:45 -05:00
sushilicious	30555b576c	Added commit mode The commit mode ensures pictures are not corrupted when Instaloader is unexpectedly interrupted. In the case that the last picture is corrupted because of an interruption, Instaloader will redownload the picture. Since the metadata is the last object saved to disk, we can consider a post as "committed" if its json metadata file exists and is not malformed. Instaloader should download any posts which are not committed. Downside is commit mode requires metadata to be saved.	2018-12-09 02:12:45 -05:00
Alexander Graf	ece0d11915	Release of version 4.1.1	2018-10-07 15:17:51 +02:00
Alexander Graf	41e734e1b8	Fix fast_update in case of no download If Instaloader was called with --fast-update and encountered a post where nothing has been downloaded because of flags that disable download of certain types of media, the download was aborted. This behavior is now fixed. Closes #183.	2018-10-07 14:59:16 +02:00
Alexander Graf	b7ea0d5953	download_profiles: Check errors only if required Fixes #178, where with `--profile-pic-only` an error message is printed (without any further implications) when downloading profile pictures of private profiles without being logged in.	2018-09-23 10:09:08 +02:00
Alexander Graf	325e89c4b2	Print how many requests were made on 429	2018-09-06 09:17:38 +02:00
Alexander Graf	e248020927	Release of version 4.1	2018-09-02 21:57:20 +02:00
Alexander Graf	5a7b76db31	Merge branch 'v4.1-dev'	2018-09-02 21:56:16 +02:00
Alexander Graf	1a239e17cb	First Release Candidate for Version 4.1	2018-08-31 17:42:59 +02:00
Alexander Graf	bb8749b753	Adjust rate control to current rate limits	2018-08-31 17:17:23 +02:00
Alexander Graf	18707ee232	docs: Better explanation for is_video Closes #173.	2018-08-30 10:16:56 +02:00
Alexander Graf	084424e856	docs/codesnippets: Link to source tree on github	2018-08-30 09:18:58 +02:00