The NodeIterator is sometimes invocated with node wrappers that are heavy to
invocate, e.g. because they trigger requests. Storing the resulting item
besides the node allows to avoid those extra requests.
In the FrozenNodeIterator, the node is still saved, rather than the item.
The GraphQL query used so far now returns edge_web_feed_timeline: null, which
raised a TypeError. Page through api/v1/feed/timeline/ instead and convert its
media items (iPhone format).
Fixes#2556
- `InstaloaderContext.get_page_data()` loads a page as a logged-out visitor and returns the query results that the page embeds. Instagram only embeds them when the request looks like a page navigation, so the request sends the `Sec-Fetch-*` headers of one.
- `Profile.from_username()` reads the profile from its public page. Anonymously, `_normalize_profile_data()` completes the node.
- Logged in, `Profile.get_posts()` uses query `28975909992013618` (`PolarisProfilePostsTabContentQuery_connection`) with the variables that the web app sends. The response has the same shape as before.
When logged in, `from_username()` does not normalize the page's node. The page shows the profile to a logged-out visitor, so normalizing would set `followed_by_viewer` to `False`, and `download_profiles()` would refuse private profiles that the session follows. Instead, the GraphQL profile query in `_obtain_metadata()` fills in the missing fields when they are first read.
Use doc_id 27128499623469141 (PolarisPostRootQuery /
xdt_api__v1__media__shortcode__web_info) which returns v1/iPhone API format. Converts the new format to the legacy GraphQL field names that the rest of the codebase expects.
Fix Profile.from_id(), Profile.from_username() and Profile._obtain_metadata() by using `users/web_profile_info/` and `users/{profile_id}/info` queries.
_mention_regex required the character before '@' to be a non-word
character other than '\n', so a mention at the start of a new line was
never matched. The '\n' exclusion was introduced in 8784ac7 when the
prefix went from '\W' to '[^\w\n]'; restoring '\W' matches mentions
after a newline while still keeping email addresses out of the results,
since the character before '@' in an address is a word character.
Also affects Profile.biography_mentions and StoryItem.caption_mentions,
which share the same regex.
Fixes#2709
When Instagram returns {"data": null} for a shortcode metadata query (e.g. under
rate limiting, or for an unavailable post), response["data"]["xdt_shortcode_media"]
raised a raw TypeError before the existing `if pic_json is None` check could run.
Guard the intermediate "data" so the clean BadResponseException is raised instead,
which the download loop already handles.
Use doc_id 26347858941511777 (search query) in from_username() to get user_id
Use doc_id 25980296051578533 (PolarisProfilePageContentQuery) in _obtain_metadata()
Add _normalize_profile_data() to map new API fields to legacy format
Handle Instagram feedback_required/checkpoint_required/challenge_required responses with a AbortDownloadException, causing the resumable iteration to be interrupted.
Co-authored-by: Alexander Graf <17130992+aandergr@users.noreply.github.com>
Alexander Graf