Compare commits

...
15 Commits
Author SHA1 Message Date
André Koch-Kramer 01c2a2b1cb Release of version 3.2.2 2018-02-15 16:13:36 +01:00
André Koch-Kramer 541b29b2e9 Remove signatures from profile pic URLs
Fixes #75 as suggested by @e5150.
2018-02-15 16:10:04 +01:00
André Koch-Kramer 6f3afd2fa1 Removed redundant character escapes in RegExp 2018-02-15 15:01:17 +01:00
André Koch-Kramer 882d460a67 Merge branch 'bug_69_fix' 2018-02-15 14:56:42 +01:00
André Koch-Kramer 9b11ca18d1 Added backward compatibility for --fast-update
This is related to bug #69.
Filenames containing the old date encoding without UTC are now
recognized when using the `--fast-update` switch.
Also non-picture files using the old date encoding should now be
recognized even when not using the `--fast-update` switch, excluding
geotags and the post metadata JSON files. The files containing comments
get renamed to the new format when touched. Caption files are only
renamed, if they get updated. Stored picture files do not get renamed at
all.
2018-01-28 16:40:13 +01:00
Alexander Graf 39bd7c55f6 doc: link next section at the end of each page
This makes it easier to go through the documentation and it
pushes readthedocs's advertising further apart from the
content.
2018-01-13 19:15:32 +01:00
André Koch-Kramer 3654d6bf6d Use UTC in filenames per default
This adresses bug #69.
All fields in the filename pattern that should be replaced by a date
encoding string (excluding {post.date_local}) are now evaluated using
UTC while ensuring that the replacing string indicates the use of UTC.
2018-01-10 22:26:49 +01:00
Alexander Graf 1ca368fa45 Mention Arch+Gentoo pkgs in doc/install section 2018-01-07 22:08:51 +01:00
Alexander Graf 698ea51cb9 Add contribution guidelines to documentation 2018-01-07 17:26:35 +01:00
Alexander Graf e6be0179b7 Support {date_utc} filename_pattern
{date_utc} encodes the post creation date in UTC rather than the
current local timezone, as {date} does.

This was proposed in #69. Encoding the post creation date in
local time zone induces problems regarding --fast-update when the
time zone is changed.
2018-01-07 14:52:03 +01:00
Alexander Graf 97e1bd2bf0 Disable Pylint's new inconsistent-return-statements 2017-12-21 14:03:44 +01:00
Alexander Graf 5695bca9f3 Current version and release date in doc sidebar 2017-12-21 13:53:06 +01:00
Alexander Graf 75d9dda4ba --no-video-thumbnails option
Closes #61.
2017-12-14 19:16:39 +01:00
Alexander Graf 41a50ffa5a Refactor download_story() into a method 2017-12-14 16:17:46 +01:00
Alexander Graf 3daaa6fbbc --no-profile-pic to disable profile pic download
(Suggested in #61)
2017-12-14 15:55:53 +01:00
12 changed files with 338 additions and 97 deletions
+1 -1
View File
@@ -9,7 +9,7 @@ python:
install:
- pip install pylint requests
script:
- python3 -m pylint -r n -d bad-whitespace,missing-docstring,too-many-arguments,locally-disabled,line-too-long,too-many-public-methods,too-many-lines,too-many-instance-attributes,too-many-locals,too-many-branches,too-many-statements instaloader
- python3 -m pylint -r n -d bad-whitespace,missing-docstring,too-many-arguments,locally-disabled,line-too-long,too-many-public-methods,too-many-lines,too-many-instance-attributes,too-many-locals,too-many-branches,too-many-statements,inconsistent-return-statements instaloader
deploy:
provider: pypi
user: aandergr
+1 -1
View File
@@ -10,6 +10,6 @@ https://instaloader.readthedocs.io/
The documentation is created with [Sphinx](http://www.sphinx-doc.org/). To build it, use
```
pip3 install sphinx sphinx-autodoc-typehints
pip3 install -r requirements.txt
make html
```
+5
View File
@@ -4,6 +4,11 @@
<li class="toctree-l1"><a class="reference internal" href="{{ next.link|e }}">{{ next.title }}</a></li>
</ul>
{% endif %}
<h3>Current Release</h3>
<ul>
<li class="toctree-l1"><a href="https://github.com/Thammus/instaloader/releases/tag/v{{current_release}}">Version <b>{{current_release}}</b></a></li>
<li>Released on <b>{{current_release_date}}</b>.</li>
</ul>
<h3>Links</h3>
<ul>
<li><a href="https://github.com/Thammus/instaloader/issues">Issue Tracker</a></li>
+10
View File
@@ -0,0 +1,10 @@
{% extends "!page.html" %}
{% block body %}
{{ super() }}
{% if next and prev %}
<h2>Next Section</h2>
<p>
<a class="reference internal" href="{{ next.link|e }}">{{ next.title }}</a>
</p>
{% endif %}
{% endblock %}
+1 -1
View File
@@ -95,7 +95,7 @@ pattern, the token ``{target}`` is replaced by the target name, and
:option:`--filename-pattern` configures the path of the post's files relative
to the target directory. The default is ``--filename-pattern={date}``.
The tokens ``{target}`` and ``{profile}`` are replaced like in the
dirname pattern. Further, the tokens ``{date}`` and ``{shortcode}`` are
dirname pattern. Further, the tokens ``{date}``, ``{date_utc}`` and ``{shortcode}`` are
defined. Additionally, in case of not downloading stories, the attributes of
:class:`.Post` can be used, e.g. ``{post.owner_id}`` or ``{post.mediaid}``.
+11 -2
View File
@@ -26,10 +26,18 @@ it by its unique ID** and renames the folder likewise.
Only download profile picture.
.. option:: --no-profile-pic
Do not download profile picture.
.. option:: --no-videos
Do not download videos.
.. option:: --no-video-thumbnails
Do not download thumbnails of videos.
.. option:: --geotags
**Download geotags** when available. Geotags are stored as a text file with
@@ -68,7 +76,8 @@ it by its unique ID** and renames the folder likewise.
.. option:: --stories-only
Rather than downloading regular posts of each specified profile, only
download stories. Requires :option:`--login`.
download stories. Requires :option:`--login`. Does not imply
:option:`--no-profile-pic`.
.. option:: --only-if filter
@@ -135,7 +144,7 @@ How to Download
Prefix of filenames. Posts are stored in the directory whose pattern is given
with ``--dirname-pattern``. ``{profile}`` is replaced by the profile name,
``{target}`` is replaced by the target you specified, i.e. either ``:feed``,
``#hashtag`` or the profile name. Also, the fields ``{date}`` and
``#hashtag`` or the profile name. Also, the fields ``{date}``, ``{date_utc}`` and
``{shortcode}`` can be specified. In case of not downloading stories, the
attributes of the :class:`.Post` class can be used in addition, e.g.
``{post.owner_id}`` or ``{post.mediaid}``.
+15
View File
@@ -19,6 +19,9 @@
#
import os
import sys
import re
import requests
from datetime import datetime
sys.path.insert(0, os.path.abspath('..'))
# -- General configuration ------------------------------------------------
@@ -352,3 +355,15 @@ texinfo_documents = [
# If true, do not generate a @detailmenu in the "Top" node's menu.
#
# texinfo_no_detailmenu = False
def get_latest_tag(repo):
tags = requests.get("https://api.github.com/repos/{}/git/refs/tags".format(repo)).json()
latest_tag = requests.get(tags[-1]['object']['url']).json()
version_string = latest_tag['tag']
version_date = datetime.strptime(latest_tag['tagger']['date'], "%Y-%m-%dT%H:%M:%SZ")
return version_string[1:], re.sub(r'\b0+(\d)', r'\1', "{:%d %b %Y}".format(version_date))
current_release, current_release_date = get_latest_tag('Thammus/instaloader')
html_context = {'current_release': current_release, 'current_release_date': current_release_date}
+83
View File
@@ -0,0 +1,83 @@
Contributing to Instaloader
===========================
As an open source project, Instaloader heavily depends on the contributions from
its community. In this document, we advise on how you may help Instaloader to
become an even greater tool.
Instaloader's development is organized on
`GitHub <https://github.com/Thammus/instaloader>`__, where Issues and Pull
Requests are discussed.
Reporting Bugs
--------------
If you encounter a bug, do not hesitate to report it in our
`Issue Tracker <https://github.com/Thammus/instaloader/issues>`__. Since
Instaloader is actively developed, the majority of bugs is fixed within only
**4 days** after being reported. When reporting a problem, please keep the
following in mind:
- Ensure you **use the latest version** of Instaloader. The currently-installed
version can be found out with ``instaloader --version``.
- Briefly **check whether the bug has already been reported**. If you find an
issue reporting the same bug you encountered, comment there rather than
opening a new issue. However, if unsure, it is advised to create a new issue.
- State **how the bug can be reproduced**, i.e. how Instaloader was invoked
when the problem occured (of course, you may anonymize profile names etc.).
- Include all **error messages and tracebacks** in the report.
- If not obvious, describe **which behavior you expected**
instead of what actually happened.
- If you think an issue has been closed accidentally or inappropriately, feel
free to reopen it.
Writing Code or Improving the Documentation
-------------------------------------------
Changes of the Instaloader source can be proposed as a
`Pull Request <https://github.com/Thammus/instaloader/pulls>`__. There are only
few things to consider:
- Sometimes, the most current code is not in the ``master`` branch. Check that
you forked from the most recent branch.
- We use `Pylint <https://www.pylint.org/>`__ for error and syntax checking of
``instaloader.py``. The file ``.travis.yml`` in the project's root directory
shows how it is invoked. Note that sometimes it might be better to disable a
warning rather than adapting the code to Pylint's desires.
- The documentation source is located in the ``docs`` folder. The file
``cli-options.rst`` is merely an RST-formatted copy of ``ìnstaloader --help``
output, of which the source is near the bottom of ``instaloader.py``.
- Feel free to contact us, even if you "only" have Proof-of-Concepts or
not-fully integrated changes. They already might be an advance for the
project.
Suggesting Features
-------------------
If you have an idea of how Instaloader should be enhanced, but do not want to
implement the feature yourself, feel free to open a ticket in our
`Issue Tracker <https://github.com/Thammus/instaloader/issues>`__.
Please consider the following:
- Instaloader already has plenty of features. **Check the documentation**
beforehand to ensure your desired suggestion is not already implemented.
- Briefly **ensure that your idea has not already been suggested**. If you find
an issue suggesting the same or a similar feature, share your thoughts in a
comment there, instead of opening a new issue.
- If possible, provide us a **use case of the feature**: How could the user
invoke the new function? Which problem would it solve? If new information is
obtained, how would it be further processed?
- **Be patient**. Naturally, bugs and pull requests have a higher priority than
feature suggestions. Keep in mind that this is a free software project, and
unfortunately we only have limited time to work on it.
+1
View File
@@ -51,6 +51,7 @@ Table of Contents
basic-usage
cli-options
as-module
contributing
Useful Links
------------
+12 -4
View File
@@ -24,7 +24,15 @@ Instaloader requires
`requests <http://python-requests.org/>`__, which
will be installed automatically, if it is not already installed.
If you do not want to use pip, even though it is highly recommended,
and prefer installing Instaloader manually,
`Download the Source <https://github.com/Thammus/instaloader/releases>`__,
extract the Zip or Tarball and execute ``instaloader.py`` from there.
**Alternative methods for installing Instaloader:**
- If you do not want to use pip, even though it is highly recommended,
and prefer to **install Instaloader manually**,
`Download the Source <https://github.com/Thammus/instaloader/releases>`__,
extract the Zip or Tarball and execute ``instaloader.py`` from there.
- On **Arch Linux**, you may install Instaloader using the
`Instaloader AUR package <https://aur.archlinux.org/packages/instaloader/>`__.
- On **Gentoo Linux**, you may install Instaloader using the
`Instaloader Ebuild <https://bugs.gentoo.org/642100>`__.
+2
View File
@@ -1 +1,3 @@
requests
sphinx
sphinx-autodoc-typehints
+196 -88
View File
@@ -29,7 +29,7 @@ import requests.utils
import urllib3
__version__ = '3.2.1'
__version__ = '3.2.2'
try:
@@ -53,6 +53,10 @@ class QueryReturnedNotFoundException(InstaloaderException):
pass
class QueryReturnedForbiddenException(InstaloaderException):
pass
class ProfileNotExistsException(InstaloaderException):
pass
@@ -267,10 +271,15 @@ class Post:
return int(self._field('owner', 'id'))
@property
def date(self) -> datetime:
"""Timestamp when the post was created."""
def date_local(self) -> datetime:
"""Timestamp when the post was created (local time zone)."""
return datetime.fromtimestamp(self._node["date"] if "date" in self._node else self._node["taken_at_timestamp"])
@property
def date_utc(self) -> datetime:
"""Timestamp when the post was created (UTC)."""
return datetime.utcfromtimestamp(self._node["date"] if "date" in self._node else self._node["taken_at_timestamp"])
@property
def url(self) -> str:
"""URL of the picture / video thumbnail of the post"""
@@ -416,6 +425,7 @@ class Instaloader:
dirname_pattern: Optional[str] = None,
filename_pattern: Optional[str] = None,
download_videos: Tristate = Tristate.always,
download_video_thumbnails: Tristate = Tristate.always,
download_geotags: Tristate = Tristate.no_extra_query,
save_captions: Tristate = Tristate.no_extra_query,
download_comments: Tristate = Tristate.no_extra_query,
@@ -429,9 +439,19 @@ class Instaloader:
self.sleep = sleep
self.quiet = quiet
self.dirname_pattern = dirname_pattern if dirname_pattern is not None else '{target}'
self.filename_pattern = filename_pattern.replace('{date}', '{date:%Y-%m-%d_%H-%M-%S}') \
if filename_pattern is not None else '{date:%Y-%m-%d_%H-%M-%S}'
if filename_pattern is not None:
filename_pattern = re.sub(r"({(?:post\.)?date)([:}])", r"\1_utc\2", filename_pattern)
self.filename_pattern_old = filename_pattern.replace('{date_utc}', '{date_utc:%Y-%m-%d_%H-%M-%S}')
self.filename_pattern_old = re.sub(r"(?i)({(?:post\.)?date_utc:[^}]*?)_UTC",
r"\1", self.filename_pattern_old)
filename_pattern = re.sub(r"(?i)({(date_utc|post\.date_utc):(?![^}]*UTC[^}]*).*?)}",
r"\1_UTC}", filename_pattern)
self.filename_pattern = filename_pattern.replace('{date_utc}', '{date_utc:%Y-%m-%d_%H-%M-%S_UTC}')
else:
self.filename_pattern = '{date_utc:%Y-%m-%d_%H-%M-%S_UTC}'
self.filename_pattern_old = '{date_utc:%Y-%m-%d_%H-%M-%S}'
self.download_videos = download_videos
self.download_video_thumbnails = download_video_thumbnails
self.download_geotags = download_geotags
self.save_captions = save_captions
self.download_comments = download_comments
@@ -454,7 +474,9 @@ class Instaloader:
"""Yield an anonymous, otherwise equally-configured copy of an Instaloader instance; Then copy its error log."""
new_loader = Instaloader(self.sleep, self.quiet, self.user_agent,
self.dirname_pattern, self.filename_pattern,
self.download_videos, self.download_geotags,
self.download_videos,
self.download_video_thumbnails,
self.download_geotags,
self.save_captions, self.download_comments,
self.save_metadata, self.max_connection_attempts)
new_loader.previous_queries = self.previous_queries
@@ -507,6 +529,9 @@ class Instaloader:
resp.raw.decode_content = True
shutil.copyfileobj(resp.raw, file)
else:
if resp.status_code == 403:
# suspected invalid URL signature
raise QueryReturnedForbiddenException("403 when accessing {}.".format(url))
if resp.status_code == 404:
# 404 not worth retrying.
raise QueryReturnedNotFoundException("404 when accessing {}.".format(url))
@@ -712,17 +737,24 @@ class Instaloader:
lambda d: d['data']['user']['edge_follow'])
def download_pic(self, filename: str, url: str, mtime: datetime,
filename_suffix: Optional[str] = None) -> bool:
filename_alt: Optional[str] = None, filename_suffix: Optional[str] = None) -> bool:
"""Downloads and saves picture with given url under given directory with given timestamp.
Returns true, if file was actually downloaded, i.e. updated."""
urlmatch = re.search('\\.[a-z0-9]*\\?', url)
file_extension = url[-3:] if urlmatch is None else urlmatch.group(0)[1:-1]
if filename_suffix is not None:
filename += '_' + filename_suffix
if filename_alt is not None:
filename_alt += '_' + filename_suffix
filename += '.' + file_extension
if os.path.isfile(filename):
self._log(filename + ' exists', end=' ', flush=True)
return False
if filename_alt is not None:
filename_alt += '.' + file_extension
if os.path.isfile(filename_alt):
self._log(filename_alt + 'exists', end=' ', flush=True)
return False
self._get_and_write_raw(url, filename)
os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))
return True
@@ -733,15 +765,20 @@ class Instaloader:
json.dump(post, fp=open(filename, 'w'), indent=4, default=Post.json_encoder)
self._log('json', end=' ', flush=True)
def update_comments(self, filename: str, post: Post) -> None:
filename += '_comments.json'
def update_comments(self, filename: str, post: Post, filename_alt: Optional[str] = None) -> None:
try:
comments = json.load(open(filename))
filename_current = filename + '_comments.json'
comments = json.load(open(filename_current))
except FileNotFoundError:
comments = list()
try:
filename_current = filename_alt + '_comments.json'
comments = json.load(open(filename_current))
except (FileNotFoundError, TypeError):
filename_current = filename + '_comments.json'
comments = list()
comments.extend(post.get_comments())
if comments:
with open(filename, 'w') as file:
with open(filename_current, 'w') as file:
comments_list = sorted(sorted(list(comments), key=lambda t: t['id']),
key=lambda t: t['created_at'], reverse=True)
unique_comments_list = [comments_list[0]]
@@ -754,17 +791,25 @@ class Instaloader:
if x['id'] != y['id']:
unique_comments_list.append(y)
file.write(json.dumps(unique_comments_list, indent=4))
os.rename(filename_current, filename + '_comments.json')
self._log('comments', end=' ', flush=True)
def save_caption(self, filename: str, mtime: datetime, caption: str) -> None:
def save_caption(self, filename: str, mtime: datetime, caption: str, filename_alt: Optional[str] = None) -> None:
"""Updates picture caption"""
filename += '.txt'
if filename_alt is not None:
filename_alt += '.txt'
pcaption = caption.replace('\n', ' ').strip()
caption = caption.encode("UTF-8")
pcaption = '[' + ((pcaption[:29] + u"\u2026") if len(pcaption) > 31 else pcaption) + ']'
with suppress(FileNotFoundError):
with open(filename, 'rb') as file:
file_caption = file.read()
try:
with open(filename, 'rb') as file:
file_caption = file.read()
except FileNotFoundError:
if filename_alt is not None:
with open(filename_alt, 'rb') as file:
file_caption = file.read()
if file_caption.replace(b'\r\n', b'\n') == caption.replace(b'\r\n', b'\n'):
try:
self._log(pcaption + ' unchanged', end=' ', flush=True)
@@ -772,15 +817,22 @@ class Instaloader:
self._log('txt unchanged', end=' ', flush=True)
return None
else:
def get_filename(index):
return filename if index == 0 else (filename[:-4] + '_old_' +
(str(0) if index < 10 else str()) + str(index) + filename[-4:])
def get_filename(file, index):
return file if index == 0 else (file[:-4] + '_old_' +
(str(0) if index < 10 else str()) + str(index) + file[-4:])
i = 0
while os.path.isfile(get_filename(i)):
file_exists_list = []
while True:
file_exists_list.append(1 if os.path.isfile(get_filename(filename, i)) else 0)
if not file_exists_list[i] and filename_alt is not None:
file_exists_list[i] = 2 if os.path.isfile(get_filename(filename_alt, i)) else 0
if not file_exists_list[i]:
break
i = i + 1
for index in range(i, 0, -1):
os.rename(get_filename(index - 1), get_filename(index))
os.rename(get_filename(filename if file_exists_list[index - 1] % 2 else filename_alt, index - 1),
get_filename(filename, index))
try:
self._log(pcaption + ' updated', end=' ', flush=True)
except UnicodeEncodeError:
@@ -803,8 +855,11 @@ class Instaloader:
os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))
self._log('geo', end=' ', flush=True)
def download_profilepic(self, name: str, url: str) -> None:
"""Downloads and saves profile pic with given url."""
def download_profilepic(self, name: str, profile_metadata: Dict[str, Any]) -> None:
"""Downloads and saves profile pic."""
url = profile_metadata["user"]["profile_pic_url_hd"] if "profile_pic_url_hd" in profile_metadata["user"] \
else profile_metadata["user"]["profile_pic_url"]
def _epoch_to_string(epoch: datetime) -> str:
return epoch.strftime('%Y-%m-%d_%H-%M-%S')
@@ -822,8 +877,13 @@ class Instaloader:
if os.path.isfile(filename):
self._log(filename + ' already exists')
return None
url = re.sub(r'/s([1-9][0-9]{2})x\1/', '/s2048x2048/', url)
self._get_and_write_raw(url, filename)
url_best = re.sub(r'/s([1-9][0-9]{2})x\1/', '/s2048x2048/', url)
url_best = re.sub(r'/vp/[a-f0-9]{32}/[A-F0-9]{8}/', '/', url_best) # remove signature
try:
self._get_and_write_raw(url_best, filename)
except (QueryReturnedForbiddenException, QueryReturnedNotFoundException) as err:
self.error('{} Retrying with lower quality version.'.format(err))
self._get_and_write_raw(url, filename)
os.utime(filename, (datetime.now().timestamp(), date_object.timestamp()))
self._log('') # log output of _get_and_write_raw() does not produce \n
@@ -903,53 +963,68 @@ class Instaloader:
profilename = post.owner_username if needs_profilename else None
dirname = self.dirname_pattern.format(profile=profilename, target=target.lower())
filename = dirname + '/' + self.filename_pattern.format(profile=profilename, target=target.lower(),
date=post.date, shortcode=post.shortcode,
date_utc=post.date_utc,
shortcode=post.shortcode,
post=post)
filename_old = dirname + '/' + self.filename_pattern_old.replace("{post.date_utc", "{date_utc") \
.format(profile=profilename, target=target.lower(),
date_utc=post.date_local,
shortcode=post.shortcode,
post=post)
os.makedirs(os.path.dirname(filename), exist_ok=True)
# Download the image(s) / video thumbnail and videos within sidecars if desired
downloaded = False
if post.typename == 'GraphSidecar':
edge_number = 1
downloaded = True
for edge in post.get_sidecar_edges():
edge_downloaded = self.download_pic(filename=filename,
# Download picture or video thumbnail
if not edge['node']['is_video'] or self.download_video_thumbnails is Tristate.always:
downloaded |= self.download_pic(filename=filename,
filename_alt=filename_old,
url=edge['node']['display_url'],
mtime=post.date,
mtime=post.date_local,
filename_suffix=str(edge_number))
# Additionally download video if available and desired
if edge['node']['is_video'] and self.download_videos is Tristate.always:
self.download_pic(filename=filename,
url=edge['node']['video_url'],
mtime=post.date,
filename_suffix=str(edge_number))
downloaded = downloaded and edge_downloaded
downloaded |= self.download_pic(filename=filename,
filename_alt=filename_old,
url=edge['node']['video_url'],
mtime=post.date_local,
filename_suffix=str(edge_number))
edge_number += 1
elif post.typename in ['GraphImage', 'GraphVideo']:
downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date)
elif post.typename == 'GraphImage':
downloaded = self.download_pic(filename=filename, filename_alt=filename_old,
url=post.url, mtime=post.date_local)
elif post.typename == 'GraphVideo':
if self.download_video_thumbnails is Tristate.always:
downloaded = self.download_pic(filename=filename, filename_alt=filename_old,
url=post.url, mtime=post.date_local)
else:
self.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))
downloaded = False
# Save caption if desired
if self.save_captions is not Tristate.never:
if post.caption:
self.save_caption(filename, post.date, post.caption)
self.save_caption(filename=filename, filename_alt=filename_old,
mtime=post.date_local, caption=post.caption)
else:
self._log("<no caption>", end=' ', flush=True)
# Download video if desired
if post.is_video and self.download_videos is Tristate.always:
self.download_pic(filename=filename, url=post.video_url, mtime=post.date)
downloaded |= self.download_pic(filename=filename, filename_alt=filename_old,
url=post.video_url, mtime=post.date_local)
# Download geotags if desired
if self.download_geotags is Tristate.always:
location = post.get_location()
if location:
self.save_location(filename, location, post.date)
self.save_location(filename, location, post.date_local)
# Update comments if desired
if self.download_comments is Tristate.always:
self.update_comments(filename, post)
self.update_comments(filename=filename, filename_alt=filename_old, post=post)
# Save metadata as JSON if desired. It might require an extra query, depending on which information has been
# already obtained. Regarding Tristate interpretation, we always assume that it requires an extra query.
@@ -1025,44 +1100,61 @@ class Instaloader:
for item in user_stories["items"]:
self._log("[%3i/%3i] " % (count, totalcount), end="", flush=True)
count += 1
shortcode = item["code"] if "code" in item else "no_code"
date = datetime.fromtimestamp(item["taken_at"])
dirname = self.dirname_pattern.format(profile=name, target=filename_target)
filename = dirname + '/' + self.filename_pattern.format(profile=name, target=filename_target,
date=date,
shortcode=shortcode)
os.makedirs(os.path.dirname(filename), exist_ok=True)
with self._error_catcher('Download story {} from user {}'.format(shortcode, name)):
if "image_versions2" in item:
url = item["image_versions2"]["candidates"][0]["url"]
downloaded = self.download_pic(filename=filename,
url=url,
mtime=date)
else:
self._log("Warning: Unable to find story image.")
downloaded = False
if "caption" in item and item["caption"] is not None and \
self.save_captions is not Tristate.never:
caption = item["caption"]
if isinstance(caption, dict) and "text" in caption:
caption = caption["text"]
self.save_caption(filename, date, caption)
else:
self._log("<no caption>", end=' ', flush=True)
if "video_versions" in item and self.download_videos is Tristate.always:
downloaded = self.download_pic(filename=filename,
url=item["video_versions"][0]["url"],
mtime=date)
if item["story_locations"] and self.download_geotags is not Tristate.never:
location = item["story_locations"][0]["location"]
if location:
self.save_location(filename, location, date)
self._log()
with self._error_catcher('Download story from user {}'.format(name)):
downloaded = self.download_story(item, filename_target, name)
if fast_update and not downloaded:
break
def download_story(self, item: Dict[str, Any], target: str, profile: str) -> bool:
"""Download one user story.
:param item: Story item, as in story['items'] for story in :meth:`get_stories`
:param target: Replacement for {target} in dirname_pattern and filename_pattern
:param profile: Owner profile name
:return: True if something was downloaded, False otherwise, i.e. file was already there
"""
shortcode = item["code"] if "code" in item else "no_code"
date_local = datetime.fromtimestamp(item["taken_at"])
date_utc = datetime.utcfromtimestamp(item["taken_at"])
dirname = self.dirname_pattern.format(profile=profile, target=target)
filename = dirname + '/' + self.filename_pattern.format(profile=profile, target=target,
date_utc=date_utc,
shortcode=shortcode)
filename_old = dirname + '/' + self.filename_pattern_old.format(profile=profile, target=target,
date_utc=date_local,
shortcode=shortcode)
os.makedirs(os.path.dirname(filename), exist_ok=True)
downloaded = False
if "image_versions2" in item:
if "video_versions" not in item or self.download_video_thumbnails is Tristate.always:
url = item["image_versions2"]["candidates"][0]["url"]
downloaded = self.download_pic(filename=filename,
filename_alt=filename_old,
url=url,
mtime=date_local)
else:
self._log("Warning: Unable to find story image.")
if "caption" in item and item["caption"] is not None and \
self.save_captions is not Tristate.never:
caption = item["caption"]
if isinstance(caption, dict) and "text" in caption:
caption = caption["text"]
self.save_caption(filename=filename, filename_alt=filename_old, mtime=date_local, caption=caption)
else:
self._log("<no caption>", end=' ', flush=True)
if "video_versions" in item and self.download_videos is Tristate.always:
downloaded |= self.download_pic(filename=filename,
filename_alt=filename_old,
url=item["video_versions"][0]["url"],
mtime=date_local)
if item["story_locations"] and self.download_geotags is not Tristate.never:
location = item["story_locations"][0]["location"]
if location:
self.save_location(filename, location, date_local)
self._log()
return downloaded
def get_feed_posts(self) -> Iterator[Post]:
"""Get Posts of the user's feed."""
@@ -1240,7 +1332,8 @@ class Instaloader:
end_cursor = media['page_info']['end_cursor']
def download_profile(self, name: str,
profile_pic_only: bool = False, fast_update: bool = False,
profile_pic: bool = True, profile_pic_only: bool = False,
fast_update: bool = False,
download_stories: bool = False, download_stories_only: bool = False,
filter_func: Optional[Callable[[Post], bool]] = None) -> None:
"""Download one profile"""
@@ -1261,8 +1354,9 @@ class Instaloader:
profile_metadata = self.get_profile_metadata(name)
# Download profile picture
with self._error_catcher('Download profile picture of {}'.format(name)):
self.download_profilepic(name, profile_metadata["user"]["profile_pic_url"])
if profile_pic or profile_pic_only:
with self._error_catcher('Download profile picture of {}'.format(name)):
self.download_profilepic(name, profile_metadata)
if profile_pic_only:
return
@@ -1320,7 +1414,8 @@ class Instaloader:
def main(self, profilelist: List[str], username: Optional[str] = None, password: Optional[str] = None,
sessionfile: Optional[str] = None, max_count: Optional[int] = None,
profile_pic_only: bool = False, fast_update: bool = False,
profile_pic: bool = True, profile_pic_only: bool = False,
fast_update: bool = False,
stories: bool = False, stories_only: bool = False,
filter_str: Optional[str] = None) -> None:
"""Download set of profiles, hashtags etc. and handle logging in and session files if desired."""
@@ -1384,7 +1479,7 @@ class Instaloader:
for target in targets:
with self._error_catcher():
try:
self.download_profile(target, profile_pic_only, fast_update, stories, stories_only,
self.download_profile(target, profile_pic, profile_pic_only, fast_update, stories, stories_only,
filter_func=filter_func)
except ProfileNotExistsException as err:
if username is not None:
@@ -1392,8 +1487,8 @@ class Instaloader:
self._log("Trying again anonymously, helps in case you are just blocked.")
with self.anonymous_copy() as anonymous_loader:
with self._error_catcher():
anonymous_loader.download_profile(target, profile_pic_only, fast_update,
filter_func=filter_func)
anonymous_loader.download_profile(target, profile_pic, profile_pic_only,
fast_update, filter_func=filter_func)
else:
raise err
except KeyboardInterrupt:
@@ -1428,8 +1523,12 @@ def main():
'followees.')
g_what.add_argument('-P', '--profile-pic-only', action='store_true',
help='Only download profile picture.')
g_what.add_argument('--no-profile-pic', action='store_true',
help='Do not download profile picture.')
g_what.add_argument('-V', '--no-videos', action='store_true',
help='Do not download videos.')
g_what.add_argument('--no-video-thumbnails', action='store_true',
help='Do not download thumbnails of videos.')
g_what.add_argument('-G', '--geotags', action='store_true',
help='Download geotags when available. Geotags are stored as a '
'text file with the location\'s name and a Google Maps link. '
@@ -1450,7 +1549,7 @@ def main():
help='Also download stories of each profile that is downloaded. Requires --login.')
g_what.add_argument('--stories-only', action='store_true',
help='Rather than downloading regular posts of each specified profile, only download '
'stories. Requires --login.')
'stories. Requires --login. Does not imply --no-profile-pic.')
g_what.add_argument('--only-if', metavar='filter',
help='Expression that, if given, must evaluate to True for each post to be downloaded. Must be '
'a syntactically valid python expression. Variables are evaluated to '
@@ -1523,6 +1622,7 @@ def main():
"eventually --only-if=viewer_has_liked.")
download_videos = Tristate.always if not args.no_videos else Tristate.no_extra_query
download_video_thumbnails = Tristate.always if not args.no_video_thumbnails else Tristate.never
download_comments = Tristate.always if args.comments else Tristate.no_extra_query
save_captions = Tristate.no_extra_query if not args.no_captions else Tristate.never
save_metadata = Tristate.always if args.metadata_json else Tristate.never
@@ -1539,13 +1639,21 @@ def main():
loader = Instaloader(sleep=not args.no_sleep, quiet=args.quiet,
user_agent=args.user_agent,
dirname_pattern=args.dirname_pattern, filename_pattern=args.filename_pattern,
download_videos=download_videos, download_geotags=download_geotags,
download_videos=download_videos, download_video_thumbnails=download_video_thumbnails,
download_geotags=download_geotags,
save_captions=save_captions, download_comments=download_comments,
save_metadata=save_metadata, max_connection_attempts=args.max_connection_attempts)
loader.main(args.profile, args.login.lower() if args.login is not None else None, args.password,
args.sessionfile,
int(args.count) if args.count is not None else None,
args.profile_pic_only, args.fast_update, args.stories, args.stories_only, args.only_if)
loader.main(args.profile,
username=args.login.lower() if args.login is not None else None,
password=args.password,
sessionfile=args.sessionfile,
max_count=int(args.count) if args.count is not None else None,
profile_pic=not args.no_profile_pic,
profile_pic_only=args.profile_pic_only,
fast_update=args.fast_update,
stories=args.stories,
stories_only=args.stories_only,
filter_str=args.only_if)
except InstaloaderException as err:
raise SystemExit("Fatal error: %s" % err)