Release of version 1.3

Fixes #19.

.travis.yml

@@ -1,11 +1,12 @@
 language: python
 python:
- - "3.3"
- - "3.4"
  - "3.5"
  - "3.5-dev"
+ - "3.6"
+ - "3.6-dev"
+ - "3.7-dev"
  - "nightly"
 install:
  - pip install pylint requests
 script:
- - python3 -m pylint -r n -d bad-whitespace,bad-continuation,missing-docstring,multiple-imports,too-many-arguments,locally-disabled instaloader
+ - python3 -m pylint -r n -d bad-whitespace,bad-continuation,missing-docstring,multiple-imports,too-many-arguments,locally-disabled,line-too-long instaloader

README.md

@@ -1,103 +0,0 @@
-# instaloader
-
-Tool to download pictures (or videos) and captions from Instagram, from a given set
-of profiles, from your feed or from all followees of a given profile.
-
-## Usage
-
-instaloader is written in Python, thus ensure having
-[Python](https://www.python.org/) (at least version 3.3) installed.
-
-If you intend to use this tool under Windows, it is highly recommended to first install
-[win-unicode-console](https://pypi.python.org/pypi/win_unicode_console).
-
-After having [downloaded instaloader](https://github.com/Thammus/instaloader/releases), unzip it and
-invoke bundled `setup.py` (requiring [setuptools](https://pypi.python.org/pypi/setuptools)) to
-install it:
-```
-./setup.py install [--user]
-```
-(pass `--user` to install it for your user only instead of globally)
-instaloader requires [python3-requests](https://pypi.python.org/pypi/requests/), which will be
-installed automatically by setup.py, if not already installed.
-
-Now, to download a set of profiles, do
-```
-instaloader profile [profile ...]
-```
-where `profile` is the name of a profile you want to download. Instead of only one profile, you may
-also specify a list of profiles.
-
-To later update your local copy of that profile, you may run
-```
-instaloader --fast-update profile [profile ...]
-```
-When `--fast-update` is given, instaloder terminates when arriving at the first already-downloaded
-picture.
-
-Instaloader can also be used to **download private profiles**. To do so, invoke it with
-```
-instaloader --login=your_username profile [profile ...]
-```
-When invoked like this, it also **stores the session cookies** in a file in `/tmp`, which will be
-reused later when `--login` is given. So you can download private profiles **non-interactively**
-when you already have a valid session cookies file.
-
-If you want to **download all followees of a given profile**, call
-```
-instaloader --login=your_username @profile
-```
-
-To **download all the pictures which you have liked**, call
-```
-instaloader --login=your_username :feed-liked
-```
-
-The `--quiet` option makes it also **suitable as a cron job**.
-
-To get a list of other helpful flags, run `instaloader --help`.
-
-## Usage as library
-
-You may also use parts of instaloader as library to do other interesting things.
-
-For example, to get a list of all followers of a profile as well as their follower count, do
-```python
-import instaloader
-
-# login
-session = instaloader.get_logged_in_session(USERNAME)
-
-# get followees
-followees = instaloader.get_followees(PROFILE, session)
-for f in followees:
-    print("%i\t%s\t%s" % (f['follower_count'], f['username'], f['full_name']))
-```
-
-Then, you may download all pictures of all followees with
-```python
-for f in followees:
-    try:
-        instaloader.download(f['username'], session)
-    except instaloader.NonfatalException:
-        pass
-```
-
-You could also download your last 20 liked pics with
-```python
-instaloader.download_feed_pics(session, max_count=20, fast_update=True,
-                               filter_func=lambda node: not node["likes"]["viewer_has_liked"])
-```
-
-Each Instagram profile has its own unique ID which stays unmodified even if a user changes his/her
-username. To get said ID, given the profile's name, you may call
-```python
-instaloader.get_id_by_username(PROFILE_NAME)
-```
-
-`get_followees()` also returns unique IDs for all loaded followees. To get the current username of a
-profile, given this unique ID
-`get_username_by_id()` can be used. For example:
-```python
-instaloader.get_username_by_id(session, followees[0]['id'])
-```

README.rst

@@ -0,0 +1,185 @@
+Instaloader
+===========
+
+Tool to automatically download pictures (or videos) of given profiles
+from Instagram.
+
+Installation
+------------
+
+Instaloader is written in Python, thus ensure having
+`Python <https://www.python.org/>`__ (at least version 3.5) installed.
+
+If you intend to use this tool under Windows, it is recommended
+to install
+`win-unicode-console <https://pypi.python.org/pypi/win_unicode_console>`__.
+
+If you have `pip <https://pypi.python.org/pypi/pip>`__ installed, you
+may install Instaloader using
+
+::
+
+    pip3 install instaloader
+
+Alternatively, to get the most current version of Instaloader from our
+`Git repository <https://github.com/Thammus/instaloader>`__:
+
+::
+
+    pip3 install git+https://github.com/Thammus/instaloader
+
+(pass ``--upgrade`` to upgrade if Instaloader is already installed)
+
+Instaloader requires
+`requests <https://pypi.python.org/pypi/requests>`__, which
+will be installed automatically, if it is not already installed.
+
+How to automatically download pictures from Instagram
+-----------------------------------------------------
+
+To **download all pictures and videos of a profile**, as well as the
+**profile picture**, do
+
+::
+
+    instaloader profile [profile ...]
+
+where ``profile`` is the name of a profile you want to download. Instead
+of only one profile, you may also specify a list of profiles.
+
+To later **update your local copy** of that profiles, you may run
+
+::
+
+    instaloader --fast-update profile [profile ...]
+
+When ``--fast-update`` is given, Instaloader stops when arriving at
+the first already-downloaded picture.
+
+Instaloader can also be used to **download private profiles**. To do so,
+invoke it with
+
+::
+
+    instaloader --login=your_username profile [profile ...]
+
+When invoked like this, it also **stores the session cookies** in a file
+in your temporary directory, which will be reused later when ``--login`` is given. So
+you can download private profiles **non-interactively** when you already
+have a valid session cookie file.
+
+You may also download
+**the most recent pictures by hashtag**:
+
+::
+
+    instaloader "#hashtag"
+
+If you want to **download all followees of a given profile**, call
+
+::
+
+    instaloader --login=your_username @profile
+
+To **download all the pictures from your feed which you have liked**, call
+
+::
+
+    instaloader --login=your_username :feed-liked
+
+or to **download all pictures from your feed**:
+
+::
+
+    instaloader --login=your_username :feed-all
+
+Advanced Options
+----------------
+
+The following flags can be given to Instaloader to specify how profiles should
+be downloaded.
+
+--fast-update        Stop when encountering the first already-downloaded post
+                     of a profile.
+--profile-pic-only   Only download profile pictures. Without this flag, the current
+                     profile picture and all the profile's posts are downloaded.
+--skip-videos        Skip posts which are videos.
+--geotags            Also **download geotags** and store Google Maps links in
+                     separate textfiles.
+--count COUNT        If used with ``#hashtag``, ``:feed-all`` or
+                     ``:feed-liked``: Do not attempt to download more than COUNT
+                     posts.
+--quiet              Do not output any messages except warnings and errors. This
+                     option makes Instaloader **suitable as a cron job**.
+--no-sleep           Normally, Instaloader waits a few seconds between requests
+                     to the Instagram servers. This flag inhibits this behavior.
+--password PASSWORD  If used with ``--login``, use parameter as password if no
+                     valid session file is found, instead of asking
+                     interactively.
+--sessionfile FILE   Specify an alternative place for loading and storing the
+                     session cookies. Without this flag, they are stored in a path
+                     within your temporary directory, encoding your local
+                     username and your instagram profile name.
+
+To get a list of all flags, run ``instaloader --help``.
+
+Usage as Python module
+----------------------
+
+You may also use parts of Instaloader as library to do other interesting
+things.
+
+For example, to get a list of all followees of a profile as well as
+their follower count, do
+
+.. code:: python
+
+    import instaloader
+
+    # login
+    session = instaloader.get_logged_in_session(USERNAME)
+
+    # get followees
+    followees = instaloader.get_followees(PROFILE, session)
+    for f in followees:
+        print("%i\t%s\t%s" % (f['follower_count'], f['username'], f['full_name']))
+
+Then, you may download all pictures of all followees with
+
+.. code:: python
+
+    for f in followees:
+        try:
+            instaloader.download(f['username'], session)
+        except instaloader.NonfatalException:
+            pass
+
+You could also download your last 20 liked pics with
+
+.. code:: python
+
+    instaloader.download_feed_pics(session, max_count=20, fast_update=True,
+                                   filter_func=lambda node:
+                                   not node["likes"]["viewer_has_liked"] if "likes" in node else not node["viewer_has_liked"])
+
+To download the last 20 pictures with hashtag #cat, do
+
+.. code:: python
+
+    instaloader.download_hashtag('cat', session=instaloader.get_anonymous_session(), max_count=20)
+
+Each Instagram profile has its own unique ID which stays unmodified even
+if a user changes his/her username. To get said ID, given the profile's
+name, you may call
+
+.. code:: python
+
+    instaloader.get_id_by_username(PROFILE_NAME)
+
+``get_followees()`` also returns unique IDs for all loaded followees. To
+get the current username of a profile, given this unique ID
+``get_username_by_id()`` can be used. For example:
+
+.. code:: python
+
+    instaloader.get_username_by_id(session, followees[0]['id'])

instaloader.py

@@ -1,11 +1,14 @@
 #!/usr/bin/env python3
 
 """Tool to download pictures (or videos) and captions from Instagram, from a given set
-of profiles, from your feed or from all followees of a given profile."""
+of profiles (even if private), from your feed or from all followees of a given profile."""
 
 import re, json, datetime, shutil, os, time, random, sys, pickle, getpass, tempfile
 from argparse import ArgumentParser
 from io import BytesIO
+from numbers import Real
+from typing import List, Optional, Any, Dict, Callable
+
 import requests, requests.utils
 
 # To get version from setup.py for instaloader --version
@@ -37,6 +40,9 @@ class NonfatalException(InstaloaderException):
 class ProfileNotExistsException(NonfatalException):
     pass
 
+class ProfileAccessDeniedException(NonfatalException):
+    pass
+
 class ProfileHasNoPicsException(NonfatalException):
     pass
 
@@ -58,24 +64,25 @@ def _log(*msg, sep='', end='\n', flush=False, quiet=False):
         print(*msg, sep=sep, end=end, flush=flush)
 
 
-def get_json(name, session, max_id=0, sleep=True):
+def get_json(name: str, session: requests.Session,
+             max_id: Optional[str] = None, sleep: bool = True) -> Optional[Dict[str, Any]]:
     """Return JSON of a profile"""
-    resp = session.get('http://www.instagram.com/'+name, \
-        params={'max_id': max_id})
+    if not max_id:
+        resp = session.get('https://www.instagram.com/'+name)
+    else:
+        resp = session.get('https://www.instagram.com/'+name, params={'max_id': max_id})
     if sleep:
         time.sleep(4 * random.random() + 1)
     match = re.search('window\\._sharedData = .*<', resp.text)
-    if match is None:
-        return None
-    else:
+    if match is not None:
         return json.loads(match.group(0)[21:-2])
 
 
-def get_username_by_id(session, profile_id):
+def get_username_by_id(session: requests.Session, profile_id: int) -> str:
     """To get the current username of a profile, given its unique ID, this function can be used.
     session is required to be a logged-in (i.e. non-anonymous) session."""
     tempsession = copy_session(session)
-    tempsession.headers.update({'Content-Type' : 'application/json'})
+    tempsession.headers.update({'Content-Type' : 'application/x-www-form-urlencoded'})
     resp = tempsession.post('https://www.instagram.com/query/', data='q=ig_user(' +
                             str(profile_id) +')+%7B%0A++username%0A%7D%0A')
     if resp.status_code == 200:
@@ -86,26 +93,26 @@ def get_username_by_id(session, profile_id):
                                             "you (id: " + str(profile_id) + ").")
     else:
         if test_login(session):
-            raise ConnectionException("Username could not be determined due to connection error" +
-                                        " (id: "+ str(profile_id) +").")
+            raise ProfileAccessDeniedException("Username could not be determined due to error {0} (id: {1})."
+                                               .format(str(resp.status_code), str(profile_id)))
         raise LoginRequiredException("Login required to determine username (id: " +
                                         str(profile_id) + ").")
 
 
-def get_id_by_username(profile):
+def get_id_by_username(profile: str) -> int:
     """Each Instagram profile has its own unique ID which stays unmodified even if a user changes
     his/her username. To get said ID, given the profile's name, you may call this function."""
     data = get_json(profile, get_anonymous_session())
-    if len(data["entry_data"]) == 0 or "ProfilePage" not in data("entry_data"):
+    if "ProfilePage" not in data["entry_data"]:
         raise ProfileNotExistsException("Profile {0} does not exist.".format(profile))
     return int(data['entry_data']['ProfilePage'][0]['user']['id'])
 
 
-def _epoch_to_string(epoch):
+def _epoch_to_string(epoch: Real) -> str:
     return datetime.datetime.fromtimestamp(epoch).strftime('%Y-%m-%d_%H-%M-%S')
 
 
-def get_followees(profile, session):
+def get_followees(profile: str, session: requests.Session) -> List[Dict[str, Any]]:
     """
     Retrieve list of followees of given profile
 
@@ -138,7 +145,7 @@ def get_followees(profile, session):
              "&ref=relationships%3A%3Afollow_list"]
     tmpsession.headers.update(default_http_header())
     tmpsession.headers.update({'Referer' : 'https://www.instagram.com/'+profile+'/following/'})
-    tmpsession.headers.update({'Content-Type' : 'application/json'})
+    tmpsession.headers.update({'Content-Type' : 'application/x-www-form-urlencoded'})
     resp = tmpsession.post('https://www.instagram.com/query/', data=query[0]+"first("+query[1])
     if resp.status_code == 200:
         data = json.loads(resp.text)
@@ -162,14 +169,18 @@ def get_followees(profile, session):
     raise LoginRequiredException("Login required to gather followees.")
 
 
-def download_pic(name, url, date_epoch, outputlabel=None, quiet=False):
+def download_pic(name: str, url: str, date_epoch: Real, outputlabel: Optional[str] = None, quiet: bool = False,
+                 filename_suffix: Optional[str] = None) -> bool:
     """Downloads and saves picture with given url under given directory with given timestamp.
     Returns true, if file was actually downloaded, i.e. updated."""
     if outputlabel is None:
         outputlabel = _epoch_to_string(date_epoch)
     urlmatch = re.search('\\.[a-z]*\\?', url)
     file_extension = url[-3:] if urlmatch is None else urlmatch.group(0)[1:-1]
-    filename = name.lower() + '/' + _epoch_to_string(date_epoch) + '.' + file_extension
+    filename = name.lower() + '/' + _epoch_to_string(date_epoch)
+    if filename_suffix is not None:
+        filename += '_' + filename_suffix
+    filename += '.' + file_extension
     if os.path.isfile(filename):
         _log(outputlabel + ' exists', end=' ', flush=True, quiet=quiet)
         return False
@@ -186,7 +197,7 @@ def download_pic(name, url, date_epoch, outputlabel=None, quiet=False):
         raise ConnectionException("File \'" + url + "\' could not be downloaded.")
 
 
-def save_caption(name, date_epoch, caption, shorter_output=False, quiet=False):
+def save_caption(name: str, date_epoch: Real, caption: str, shorter_output: bool = False, quiet: bool = False) -> None:
     """Updates picture caption"""
     filename = name.lower() + '/' + _epoch_to_string(date_epoch) + '.txt'
     pcaption = caption.replace('\n', ' ').strip()
@@ -206,7 +217,7 @@ def save_caption(name, date_epoch, caption, shorter_output=False, quiet=False):
             return None
         else:
             def get_filename(index):
-                return filename if index==0 else (filename[:-4] + '_old_' + \
+                return filename if index==0 else (filename[:-4] + '_old_' +
                         (str(0) if index<10 else str()) + str(index) + filename[-4:])
             i = 0
             while os.path.isfile(get_filename(i)):
@@ -229,7 +240,19 @@ def save_caption(name, date_epoch, caption, shorter_output=False, quiet=False):
     os.utime(filename, (datetime.datetime.now().timestamp(), date_epoch))
 
 
-def download_profilepic(name, url, quiet=False):
+def save_location(name: str, location_json: Dict[str, str], date_epoch: Real, quiet: bool = False) -> None:
+    filename = name.lower() + '/' + _epoch_to_string(date_epoch) + '_location.txt'
+    location_string = location_json["name"]+"\n" + \
+                      "https://maps.google.com/maps?q={0},{1}&ll={0},{1}\n" \
+                          .format(location_json["lat"], location_json["lng"])
+    os.makedirs(name.lower(), exist_ok=True)
+    with open(filename, 'wb') as text_file:
+        shutil.copyfileobj(BytesIO(location_string.encode()), text_file)
+    os.utime(filename, (datetime.datetime.now().timestamp(), date_epoch))
+    _log('geo', end=' ', flush=True, quiet=quiet)
+
+
+def download_profilepic(name: str, url: str, quiet: bool = False) -> None:
     """Downloads and saves profile pic with given url."""
     date_object = datetime.datetime.strptime(requests.head(url).headers["Last-Modified"], \
         '%a, %d %b %Y %H:%M:%S GMT')
@@ -256,19 +279,19 @@ def download_profilepic(name, url, quiet=False):
         raise ConnectionException("File \'" + url + "\' could not be downloaded.")
 
 
-def get_default_session_filename(username):
+def get_default_session_filename(username: str) -> str:
     """Returns default session filename for given username."""
     dirname = tempfile.gettempdir() + "/" + ".instaloader-" + getpass.getuser()
     filename = dirname + "/" + "session-" + username
     return filename
 
 
-def save_session(session, username, filename=None, quiet=False):
+def save_session(session: requests.Session, username: str, filename: Optional[str] = None, quiet: bool = False) -> None:
     """Saves requests.Session object."""
     if filename is None:
         filename = get_default_session_filename(username)
     dirname = os.path.dirname(filename)
-    if not os.path.exists(dirname):
+    if dirname != '' and not os.path.exists(dirname):
         os.makedirs(dirname)
         os.chmod(dirname, 0o700)
     with open(filename, 'wb') as sessionfile:
@@ -277,7 +300,7 @@ def save_session(session, username, filename=None, quiet=False):
         _log("Saved session to %s." % filename, quiet=quiet)
 
 
-def load_session(username, filename=None, quiet=False):
+def load_session(username: str, filename: Optional[str] = None, quiet: bool = False) -> requests.Session:
     """Returns loaded requests.Session object, or None if not found."""
     if filename is None:
         filename = get_default_session_filename(username)
@@ -293,7 +316,7 @@ def load_session(username, filename=None, quiet=False):
         pass
 
 
-def copy_session(session):
+def copy_session(session: requests.Session) -> requests.Session:
     """Duplicates a requests.Session."""
     new = requests.Session()
     new.cookies = \
@@ -302,7 +325,7 @@ def copy_session(session):
     return new
 
 
-def test_login(session):
+def test_login(session: requests.Session) -> Optional[str]:
     """Returns the Instagram username to which given requests.Session object belongs, or None."""
     if session is None:
         return
@@ -313,7 +336,7 @@ def test_login(session):
     return data['config']['viewer']['username']
 
 
-def default_http_header(empty_session_only=False):
+def default_http_header(empty_session_only: bool = False) -> Dict[str, str]:
     """Returns default HTTP header we use for requests."""
     user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' \
                 '(KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36'
@@ -336,7 +359,7 @@ def default_http_header(empty_session_only=False):
     return header
 
 
-def get_anonymous_session():
+def get_anonymous_session() -> requests.Session:
     """Returns our default anonymous requests.Session object."""
     session = requests.Session()
     session.cookies.update({'sessionid' : '', 'mid' : '', 'ig_pr' : '1', \
@@ -346,7 +369,7 @@ def get_anonymous_session():
     return session
 
 
-def get_session(user, passwd):
+def get_session(user: str, passwd: str) -> requests.Session:
     """Log in to instagram with given username and password and return session object"""
     session = requests.Session()
     session.cookies.update({'sessionid' : '', 'mid' : '', 'ig_pr' : '1', \
@@ -369,7 +392,7 @@ def get_session(user, passwd):
         raise ConnectionException('Login error! Connection error!')
 
 
-def get_feed_json(session, end_cursor=None, sleep=True):
+def get_feed_json(session: requests.Session, end_cursor: str = None, sleep: bool = True) -> Dict[str, Any]:
     """
     Get JSON of the user's feed.
 
@@ -407,15 +430,28 @@ def get_feed_json(session, end_cursor=None, sleep=True):
             "profile_pic_url%2C%0A++username%0A%7D%0A&ref=feed::show"
     tmpsession.headers.update(default_http_header())
     tmpsession.headers.update({'Referer' : 'https://www.instagram.com/'})
-    tmpsession.headers.update({'Content-Type' : 'application/json'})
+    tmpsession.headers.update({'Content-Type' : 'application/x-www-form-urlencoded'})
     resp = tmpsession.post('https://www.instagram.com/query/', data=query)
     if sleep:
         time.sleep(4 * random.random() + 1)
     return json.loads(resp.text)
 
 
-def download_node(node, session, name,
-                  download_videos=True, sleep=True, shorter_output=False, quiet=False):
+def get_location(session: requests.Session, node_code: str, sleep: bool = True) -> Dict[str, str]:
+    pic_json = get_json("p/" + node_code, session, sleep=sleep)
+    media = pic_json["entry_data"]["PostPage"][0]["graphql"]["shortcode_media"] \
+        if "graphql" in pic_json["entry_data"]["PostPage"][0] \
+        else pic_json["entry_data"]["PostPage"][0]["media"]
+    if media["location"] is not None:
+        location_json = get_json("explore/locations/" +
+                                 media["location"]["id"],
+                                 session, sleep=sleep)
+        return location_json["entry_data"]["LocationsPage"][0]["location"]
+
+
+def download_node(node: Dict[str, Any], session: requests.Session, name: str,
+                  download_videos: bool = True, geotags: bool = False,
+                  sleep: bool = True, shorter_output: bool = False, quiet: bool = False) -> bool:
     """
     Download everything associated with one instagram node, i.e. picture, caption and video.
 
@@ -423,51 +459,97 @@ def download_node(node, session, name,
     :param session: Session
     :param name: Name of profile to which this node belongs
     :param download_videos: True, if videos should be downloaded
+    :param geotags: Download geotags
     :param sleep: Sleep between requests to instagram server
     :param shorter_output: Shorten log output by not printing captions
     :param quiet: Suppress output
     :return: True if something was downloaded, False otherwise, i.e. file was already there
     """
-    downloaded = download_pic(name, node["display_src"], node["date"], quiet=quiet)
-    if sleep:
-        time.sleep(1.75 * random.random() + 0.25)
-    if "caption" in node:
-        save_caption(name, node["date"], node["caption"], shorter_output, quiet)
+    # pylint:disable=too-many-branches,too-many-locals
+    date = node["date"] if "date" in node else node["taken_at_timestamp"]
+    if '__typename' in node:
+        if node['__typename'] == 'GraphSidecar':
+            sidecar_data = session.get('https://www.instagram.com/p/' + node['code'] + '/', params={'__a': 1}).json()
+            edge_number = 1
+            downloaded = True
+            media = sidecar_data["graphql"]["shortcode_media"] if "graphql" in sidecar_data else sidecar_data["media"]
+            for edge in media['edge_sidecar_to_children']['edges']:
+                edge_downloaded = download_pic(name, edge['node']['display_url'],date,
+                                               filename_suffix=str(edge_number), quiet=quiet,
+                                               outputlabel=(str(edge_number) if edge_number != 1 else None))
+                downloaded = downloaded and edge_downloaded
+                edge_number += 1
+                if sleep:
+                    time.sleep(1.75 * random.random() + 0.25)
+        elif node['__typename'] in ['GraphImage', 'GraphVideo']:
+            downloaded = download_pic(name, node["display_url"] if "display_url" in node else node["display_src"],
+                                      date, quiet=quiet)
+            if sleep:
+                time.sleep(1.75 * random.random() + 0.25)
+        else:
+            _log("Warning: Unknown typename discovered:" + node['__typename'])
+            downloaded = False
+    else:
+        # Node is an old image or video.
+        downloaded = download_pic(name, node["display_src"], date, quiet=quiet)
+        if sleep:
+            time.sleep(1.75 * random.random() + 0.25)
+    if "edge_media_to_caption" in node and node["edge_media_to_caption"]["edges"]:
+        save_caption(name, date, node["edge_media_to_caption"]["edges"][0]["node"]["text"], shorter_output, quiet)
+    elif "caption" in node:
+        save_caption(name, date, node["caption"], shorter_output, quiet)
     else:
         _log("<no caption>", end=' ', flush=True, quiet=quiet)
+    node_code = node['shortcode'] if 'shortcode' in node else node['code']
     if node["is_video"] and download_videos:
-        video_data = get_json('p/' + node["code"], session, sleep=sleep)
+        video_data = get_json('p/' + node_code, session, sleep=sleep)
         download_pic(name,
-                     video_data['entry_data']['PostPage'][0]['media']['video_url'],
-                     node["date"], 'mp4', quiet=quiet)
+                     video_data['entry_data']['PostPage'][0]['graphql']['shortcode_media']['video_url'],
+                     date, 'mp4', quiet=quiet)
+    if geotags:
+        location = get_location(session, node_code, sleep)
+        if location:
+            save_location(name, location, date, quiet=quiet)
     _log(quiet=quiet)
     return downloaded
 
 
-def download_feed_pics(session, max_count=None, fast_update=False, filter_func=None,
-                       download_videos=True, shorter_output=False, sleep=True, quiet=False):
+def download_feed_pics(session: requests.Session, max_count: int = None, fast_update: bool = False,
+                       filter_func: Optional[Callable[[Dict[str, Dict[str, Any]]], bool]] = None,
+                       download_videos: bool = True, geotags: bool = False,
+                       shorter_output: bool = False, sleep: bool = True, quiet: bool = False) -> None:
     """
     Download pictures from the user's feed.
 
     Example to download up to the 20 pics the user last liked:
     >>> download_feed_pics(load_session('USER'), max_count=20, fast_update=True,
-    >>>                    filter_func=lambda node: not node["likes"]["viewer_has_liked"])
+    >>>                    filter_func=lambda node:
+    >>>                    not node["likes"]["viewer_has_liked"] if "likes" in node else not node["viewer_has_liked"])
 
     :param session: Session belonging to a user, i.e. not an anonymous session
     :param max_count: Maximum count of pictures to download
     :param fast_update: If true, abort when first already-downloaded picture is encountered
     :param filter_func: function(node), which returns True if given picture should not be downloaded
     :param download_videos: True, if videos should be downloaded
+    :param geotags: Download geotags
     :param shorter_output: Shorten log output by not printing captions
     :param sleep: Sleep between requests to instagram server
     :param quiet: Suppress output
     """
+    # pylint:disable=too-many-locals
     data = get_feed_json(session, sleep=sleep)
     count = 1
-    while data["feed"]["media"]["page_info"]["has_next_page"]:
-        for node in data["feed"]["media"]["nodes"]:
+    while True:
+        if "graphql" in data:
+            is_edge = True
+            feed = data["graphql"]["user"]["edge_web_feed_timeline"]
+        else:
+            is_edge = False
+            feed = data["feed"]["media"]
+        for edge_or_node in feed["edges"] if is_edge else feed["nodes"]:
             if max_count is not None and count > max_count:
                 return
+            node = edge_or_node["node"] if is_edge else edge_or_node
             name = node["owner"]["username"]
             if filter_func is not None and filter_func(node):
                 _log("<pic by %s skipped>" % name, flush=True, quiet=quiet)
@@ -475,15 +557,66 @@ def download_feed_pics(session, max_count=None, fast_update=False, filter_func=N
             _log("[%3i] %s " % (count, name), end="", flush=True, quiet=quiet)
             count += 1
             downloaded = download_node(node, session, name,
-                                       download_videos=download_videos, sleep=sleep,
+                                       download_videos=download_videos, geotags=geotags,
+                                       sleep=sleep, shorter_output=shorter_output, quiet=quiet)
+            if fast_update and not downloaded:
+                return
+        if not feed["page_info"]["has_next_page"]:
+            break
+        data = get_feed_json(session, end_cursor=feed["page_info"]["end_cursor"], sleep=sleep)
+
+
+def get_hashtag_json(hashtag: str, session: requests.Session,
+                     max_id: Optional[str] = None, sleep: bool = True) -> Optional[Dict[str, Any]]:
+    """Return JSON of a #hashtag"""
+    return get_json(name='explore/tags/{0}/'.format(hashtag), session=session, max_id=max_id, sleep=sleep)
+
+
+def download_hashtag(hashtag: str, session: requests.Session,
+                     max_count: Optional[int] = None,
+                     filter_func: Optional[Callable[[Dict[str, Dict[str, Any]]], bool]] = None,
+                     fast_update: bool = False, download_videos: bool = True, geotags: bool = False,
+                     shorter_output: bool = False, sleep: bool = True, quiet: bool = False) -> None:
+    """Download pictures of one hashtag.
+
+    To download the last 30 pictures with hashtag #cat, do
+    >>> download_hashtag('cat', session=get_anonymous_session(), max_count=30)
+
+    :param hashtag: Hashtag to download, without leading '#'
+    :param session: Session belonging to a user, i.e. not an anonymous session
+    :param max_count: Maximum count of pictures to download
+    :param filter_func: function(node), which returns True if given picture should not be downloaded
+    :param fast_update: If true, abort when first already-downloaded picture is encountered
+    :param download_videos: True, if videos should be downloaded
+    :param geotags: Download geotags
+    :param shorter_output: Shorten log output by not printing captions
+    :param sleep: Sleep between requests to instagram server
+    :param quiet: Suppress output
+    """
+    data = get_hashtag_json(hashtag, session, sleep=sleep)
+    count = 1
+    while data:
+        for node in data['entry_data']['TagPage'][0]['tag']['media']['nodes']:
+            if max_count is not None and count > max_count:
+                return
+            _log('[{0:3d}] #{1} '.format(count, hashtag), end='', flush=True, quiet=quiet)
+            if filter_func is not None and filter_func(node):
+                _log('<skipped>', quiet=quiet)
+                continue
+            count += 1
+            downloaded = download_node(node, session, '#{0}'.format(hashtag),
+                                       download_videos=download_videos, geotags=geotags, sleep=sleep,
                                        shorter_output=shorter_output, quiet=quiet)
             if fast_update and not downloaded:
                 return
-        data = get_feed_json(session, end_cursor=data["feed"]["media"]["page_info"]["end_cursor"],
-                             sleep=sleep)
+        if data['entry_data']['TagPage'][0]['tag']['media']['page_info']['has_next_page']:
+            data = get_hashtag_json(hashtag, session, sleep=sleep,
+                                    max_id=data['entry_data']['TagPage'][0]['tag']['media']['page_info']['end_cursor'])
+        else:
+            break
 
 
-def check_id(profile, session, json_data, quiet=False):
+def check_id(profile: str, session: requests.Session, json_data: Dict[str, Any], quiet: bool = False) -> str:
     """
     Consult locally stored ID of profile with given name, check whether ID matches and whether name
     has changed and return current name of the profile, and store ID of profile.
@@ -520,10 +653,12 @@ def check_id(profile, session, json_data, quiet=False):
     raise ProfileNotExistsException("Profile {0} does not exist.".format(profile))
 
 
-def download(name, session, profile_pic_only=False, download_videos=True,
-        fast_update=False, shorter_output=False, sleep=True, quiet=False):
+def download(name: str, session: requests.Session,
+             profile_pic_only: bool = False, download_videos: bool = True, geotags: bool = False,
+             fast_update: bool = False, shorter_output: bool = False, sleep: bool = True,
+             quiet: bool = False) -> None:
     """Download one profile"""
-    # pylint:disable=too-many-branches
+    # pylint:disable=too-many-branches,too-many-locals
     # Get profile main page json
     data = get_json(name, session, sleep=sleep)
     # check if profile does exist or name has changed since last download
@@ -549,17 +684,13 @@ def download(name, session, profile_pic_only=False, download_videos=True,
         if data["config"]["viewer"] is not None:
             _log("profile %s could also be downloaded anonymously." % name, quiet=quiet)
     if ("nodes" not in data["entry_data"]["ProfilePage"][0]["user"]["media"] or
-            len(data["entry_data"]["ProfilePage"][0]["user"]["media"]["nodes"]) == 0) \
+            not data["entry_data"]["ProfilePage"][0]["user"]["media"]["nodes"]) \
                     and not profile_pic_only:
         raise ProfileHasNoPicsException("Profile %s: no pics found." % name)
     # Iterate over pictures and download them
     def get_last_id(data):
-        if len(data["entry_data"]) == 0 or \
-                        len(data["entry_data"]["ProfilePage"][0]["user"]["media"]["nodes"]) == 0:
-            return None
-        else:
-            data = data["entry_data"]["ProfilePage"][0]["user"]["media"]["nodes"]
-            return int(data[len(data) - 1]["id"])
+        if data["entry_data"] and data["entry_data"]["ProfilePage"][0]["user"]["media"]["nodes"]:
+            return data["entry_data"]["ProfilePage"][0]["user"]["media"]["nodes"][-1]["id"]
     totalcount = data["entry_data"]["ProfilePage"][0]["user"]["media"]["count"]
     count = 1
     while get_last_id(data) is not None:
@@ -567,13 +698,14 @@ def download(name, session, profile_pic_only=False, download_videos=True,
             _log("[%3i/%3i] " % (count, totalcount), end="", flush=True, quiet=quiet)
             count += 1
             downloaded = download_node(node, session, name,
-                                       download_videos=download_videos, sleep=sleep,
-                                       shorter_output=shorter_output, quiet=quiet)
+                                       download_videos=download_videos, geotags=geotags,
+                                       sleep=sleep, shorter_output=shorter_output, quiet=quiet)
             if fast_update and not downloaded:
                 return
         data = get_json(name, session, max_id=get_last_id(data), sleep=sleep)
 
-def get_logged_in_session(username, password=None, quiet=False):
+
+def get_logged_in_session(username: str, password: Optional[str] = None, quiet: bool = False) -> requests.Session:
     """Logs in and returns session, asking user for password if needed"""
     if password is not None:
         return get_session(username, password)
@@ -588,9 +720,12 @@ def get_logged_in_session(username, password=None, quiet=False):
             print(err, file=sys.stderr)
             password = None
 
-def download_profiles(profilelist, username=None, password=None, sessionfile=None,
-        profile_pic_only=False, download_videos=True, fast_update=False,
-        sleep=True, shorter_output=False, quiet=False):
+
+def download_profiles(profilelist: List[str], username: Optional[str] = None, password: Optional[str] = None,
+                      sessionfile: Optional[str] = None, max_count: Optional[int] = None,
+                      profile_pic_only: bool = False, download_videos: bool = True, geotags: bool = False,
+                      fast_update: bool = False,
+                      sleep: bool = True, shorter_output: bool = False, quiet: bool = False) -> None:
     """Download set of profiles and handle sessions"""
     # pylint:disable=too-many-branches,too-many-locals
     # Login, if desired
@@ -607,38 +742,52 @@ def download_profiles(profilelist, username=None, password=None, sessionfile=Non
     try:
         # Generate set of targets
         for pentry in profilelist:
-            if pentry[0] == '@':
+            if pentry[0] == '#':
+                _log("Retrieving pictures with hashtag {0}".format(pentry), quiet=quiet)
+                download_hashtag(hashtag=pentry[1:], session=session, max_count=max_count, fast_update=fast_update,
+                                 download_videos=download_videos, geotags=geotags, shorter_output=shorter_output,
+                                 sleep=sleep, quiet=quiet)
+            elif pentry[0] == '@' and username is not None:
                 _log("Retrieving followees of %s..." % pentry[1:], quiet=quiet)
                 followees = get_followees(pentry[1:], session)
                 targets.update([followee['username'] for followee in followees])
             elif pentry == ":feed-all" and username is not None:
                 _log("Retrieving pictures from your feed...", quiet=quiet)
-                download_feed_pics(session, fast_update=fast_update,
-                                   download_videos=download_videos, shorter_output=shorter_output,
-                                   sleep=sleep, quiet=quiet)
+                download_feed_pics(session, fast_update=fast_update, max_count=max_count,
+                                   download_videos=download_videos, geotags=geotags,
+                                   shorter_output=shorter_output, sleep=sleep, quiet=quiet)
             elif pentry == ":feed-liked" and username is not None:
                 _log("Retrieving pictures you liked from your feed...", quiet=quiet)
-                download_feed_pics(session, fast_update=fast_update,
-                                   filter_func=lambda node: not node["likes"]["viewer_has_liked"],
-                                   download_videos=download_videos, shorter_output=shorter_output,
-                                   sleep=sleep, quiet=quiet)
+                download_feed_pics(session, fast_update=fast_update, max_count=max_count,
+                                   filter_func=lambda node:
+                                                        not node["likes"]["viewer_has_liked"]
+                                                        if "likes" in node
+                                                        else not node["viewer_has_liked"],
+                                   download_videos=download_videos, geotags=geotags,
+                                   shorter_output=shorter_output, sleep=sleep, quiet=quiet)
             else:
                 targets.add(pentry)
-        if len(targets) == 0:
-            _log("No profiles to download given.", quiet=quiet)
-        elif len(targets) > 1:
+        if len(targets) > 1:
             _log("Downloading %i profiles..." % len(targets), quiet=quiet)
         # Iterate through targets list and download them
         for target in targets:
             try:
-                download(target, session, profile_pic_only, download_videos,
-                        fast_update, shorter_output, sleep, quiet)
+                try:
+                    download(target, session, profile_pic_only, download_videos,
+                             geotags, fast_update, shorter_output, sleep, quiet)
+                except ProfileNotExistsException as err:
+                    if username is not None:
+                        _log("\"Profile not exists\" - Trying again anonymously, helps in case you are just blocked")
+                        download(target, get_anonymous_session(), profile_pic_only, download_videos,
+                                 geotags, fast_update, shorter_output, sleep, quiet)
+                    else:
+                        raise err
             except NonfatalException as err:
                 failedtargets.append(target)
                 print(err, file=sys.stderr)
     except KeyboardInterrupt:
         print("\nInterrupted by user.", file=sys.stderr)
-    if len(targets) > 1 and len(failedtargets) > 0:
+    if len(targets) > 1 and failedtargets:
         print("Errors occured (see above) while downloading profiles: %s." %
                 ", ".join(failedtargets), file=sys.stderr)
     # Save session if it is useful
@@ -646,11 +795,13 @@ def download_profiles(profilelist, username=None, password=None, sessionfile=Non
         save_session(session, username, sessionfile, quiet=quiet)
 
 def main():
-    parser = ArgumentParser(description=__doc__)
-    parser.add_argument('profile', nargs='*',
-                        help='Name of profile to download; @<profile> to download all followees of '
+    parser = ArgumentParser(description=__doc__,
+                            epilog="Report issues at https://github.com/Thammus/instaloader/issues.")
+    parser.add_argument('profile', nargs='*', metavar='profile|#hashtag',
+                        help='Name of profile or #hashtag to download. '
+                             'Alternatively, if --login is given: @<profile> to download all followees of '
                              '<profile>; or the special targets :feed-all or :feed-liked to '
-                             'download pictures from your feed if --login is given (using '
+                             'download pictures from your feed (using '
                              '--fast-update is recommended).')
     parser.add_argument('--version', action='version',
                         version=__version__)
@@ -670,8 +821,13 @@ def main():
             help='Only download profile picture')
     parser.add_argument('-V', '--skip-videos', action='store_true',
             help='Do not download videos')
+    parser.add_argument('-G', '--geotags', action='store_true',
+            help='Store geotags when available')
     parser.add_argument('-F', '--fast-update', action='store_true',
             help='Abort at encounter of first already-downloaded picture')
+    parser.add_argument('-c', '--count',
+                        help='Do not attempt to download more than COUNT posts. '
+                             'Applies only to #hashtag, :feed-all and :feed-liked.')
     parser.add_argument('-S', '--no-sleep', action='store_true',
             help='Do not sleep between actual downloads of pictures')
     parser.add_argument('-O', '--shorter-output', action='store_true',
@@ -682,8 +838,9 @@ def main():
     args = parser.parse_args()
     try:
         download_profiles(args.profile, args.login, args.password, args.sessionfile,
-                args.profile_pic_only, not args.skip_videos, args.fast_update,
-                not args.no_sleep, args.shorter_output, args.quiet)
+                          int(args.count) if args.count is not None else None,
+                          args.profile_pic_only, not args.skip_videos, args.geotags, args.fast_update,
+                          not args.no_sleep, args.shorter_output, args.quiet)
     except InstaloaderException as err:
         raise SystemExit("Fatal error: %s" % err)
 

setup.py

@@ -1,31 +1,38 @@
 #!/usr/bin/env python3
 
+import sys
 from setuptools import setup
 
+if sys.version_info < (3, 5):
+    sys.exit('Instaloader requires Python >= 3.5.')
+
 setup(
     name='instaloader',
-    version='1.1',
+    version='1.3',
     py_modules=['instaloader'],
     url='https://github.com/Thammus/instaloader',
     license='MIT',
     author='Alexander Graf, André Koch-Kramer',
     author_email='mail@agraf.me, koch-kramer@web.de',
-    description='Tool to download pictures (or videos) and captions from Instagram, from a given '
-                'set of profiles, from your feed or from all followees of a given profile.',
+    description='Tool to download Instagram pictures (or videos) and captions from a given set of '
+                'profiles (even if private), from your feed or from all followees of a given '
+                'profile.',
+    long_description=open('README.rst').read(),
     install_requires=['requests>=2.4'],
-    python_requires='>=3.3',
+    python_requires='>=3.5',
     entry_points={'console_scripts': ['instaloader=instaloader:main']},
     zip_safe=True,
     keywords='instagram downloader',
     classifiers=[
         'Development Status :: 5 - Production/Stable',
         'Environment :: Console',
+        'Intended Audience :: End Users/Desktop',
         'License :: OSI Approved :: MIT License',
         'Operating System :: OS Independent',
-        'Programming Language :: Python :: 3.3',
-        'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
-        'Programming Language :: Python :: 3.6'
-        'Topic :: Internet'
+        'Programming Language :: Python :: 3.6',
+        'Programming Language :: Python :: 3 :: Only',
+        'Topic :: Internet',
+        'Topic :: Multimedia :: Graphics'
     ]
 )