Commit Graph

987 Commits

Author SHA1 Message Date
Eugen Rochko 36b39fbac5
Add preference to disable e-mails about new pending accounts (#10529) 2019-04-10 00:35:49 +02:00
Eugen Rochko 8b69a66380 Add "why do you want to join" field to invite requests (#10524)
* Add "why do you want to join" field to invite requests

Fix #10512

* Remove unused translations

* Fix broken registrations when no invite request text is submitted
2019-04-09 23:06:30 +09:00
ThibG cb71c95e22 Export and import show_reblogs together with following list (#10495)
* Refactor imports

* Export show_reblogs when exporting list of followed users

* Add support for importing show_reblogs with following collection

* Fix tests
2019-04-08 07:28:27 +02:00
Eugen Rochko 67b3b62b98
Improve blocked view of profiles (#10491)
* Revert "Fix filtering of favourited_by, reblogged_by, followers and following (#10447)"

This reverts commit 120544067f.

* Revert "Hide blocking accounts from blocked users (#10442)"

This reverts commit 62bafa20a1.

* Improve blocked view of profiles

- Change "You are blocked" to "Profile unavailable"
- Hide following/followers in API when blocked
- Disable follow button and show "Profile unavailable" on public profile as well
2019-04-07 04:59:13 +02:00
Eugen Rochko e1d0390e29
Add batch actions for approving and rejecting pending accounts (#10469) 2019-04-06 17:53:45 +02:00
Eugen Rochko 2c63e0292a
Fix admin validation being too strict about usernames (#10449)
* Fix admin validation being too strict about usernames

Fix #10446

* Strip Setting.site_contact_username consistently throughout the codebase
2019-04-06 17:53:17 +02:00
ThibG d4882aa64a Export and import `hide_notifications` alongside user mutes (#10335)
* Export hide_notifications along with user mutes

* Import hide_notifications along with muted users list

* Add headers for CSV exports
2019-04-03 18:17:43 +02:00
ThibG 120544067f Fix filtering of favourited_by, reblogged_by, followers and following (#10447) 2019-04-01 21:24:28 +02:00
ThibG 62bafa20a1 Hide blocking accounts from blocked users (#10442)
* Revert "Add indication that you have been blocked in web UI (#10420)"

This reverts commit bd02ec6daa.

* Revert "Add `blocked_by` relationship to the REST API (#10373)"

This reverts commit 9745de883b.

* Hide blocking accounts from search results

* Filter blocking accouts from account followers

* Filter blocking accouts from account's following accounts

* Filter blocking accounts from “reblogged by” and “favourited by” lists

* Remove blocking account from URL search

* Return 410 on trying to fetch user data from a user who blocked us

* Return 410 in /api/v1/account/statuses for suspended or blocking accounts

* Fix status filtering when performing URL search

* Restore some React improvements

Restore some cleanup from bd02ec6daa

* Refactor by adding `without_blocking` scope
2019-04-01 20:06:13 +02:00
slice 85973f4f37 Improvements to image upload validation and creation (#10431)
* Check if image value is nil? before creating an image

Check if uploaded images aren't nil before creating SiteUpload models
for them.

* Validate presence of file in SiteUpload

* Fix file presence validation

* Fabricate SiteUpload#file

* Add link to Creative Commons license
2019-04-01 07:30:46 +02:00
slice 59cce07601 Add thumbnail, hero, and mascot back to Form::AdminSettings::KEYS (#10430)
These will allow these images to be uploaded again on the admin settings
page.
2019-03-31 05:03:56 +02:00
Alex Gessner 69141dca26 squashed identity proof updates (#10375) 2019-03-28 18:01:09 +01:00
Eugen Rochko f1bc90ab50
Rename :poll to :preloadable_poll and :owned_poll to :poll on Status (#10401)
Also, fix some n+1 queries

Resolve #10365
2019-03-28 04:44:59 +01:00
Eugen Rochko e117964325
Change icons of features on admin dashboard to remove bias (#10366)
Red crosses implied that it was bad/unexpected that certain features
were not enabled. In reality, they are options, so showing a green
or grey power-off icon is more appropriate.

Add status of timeline preview as well

Fix sample accounts changing too frequently due to wrong query

Sample accounts are intended to be sorted by popularity
2019-03-26 01:24:19 +01:00
Eugen Rochko 94e5e834f2
Improve performance of list of blocked domains by caching counts (#10374) 2019-03-26 00:36:35 +01:00
Eugen Rochko 45b849bad9
Fix MergeWorker being queued for remote users (#10355) 2019-03-24 12:36:26 +01:00
Eugen Rochko 555c4e11ba
Add validations to admin settings (#10348)
* Add validations to admin settings

- Validate correct HTML markup
- Validate presence of contact username & e-mail
- Validate that all usernames are valid
- Validate that enums have expected values

* Fix code style issue

* Fix tests
2019-03-23 14:07:04 +01:00
Eugen Rochko 00d988955f
If registrations have been re-opened when user confirms account, approve (#10349) 2019-03-23 02:23:48 +01:00
ThibG 80f0910e21 Add support for custom emojis in poll options (#10322)
* Backend changes for custom emoji support in poll options

* Serialize poll emojis in REST API

* Render custom emojis in poll options

* Render custom emoji in poll options on public pages
2019-03-20 17:29:12 +01:00
Eugen Rochko 9c4cbdbafb
Add Keybase integration (#10297)
* create account_identity_proofs table

* add endpoint for keybase to check local proofs

* add async task to update validity and liveness of proofs from keybase

* first pass keybase proof CRUD

* second pass keybase proof creation

* clean up proof list and add badges

* add avatar url to keybase api

* Always highlight the “Identity Proofs” navigation item when interacting with proofs.

* Update translations.

* Add profile URL.

* Reorder proofs.

* Add proofs to bio.

* Update settings/identity_proofs front-end.

* Use `link_to`.

* Only encode query params if they exist.

URLs without params had a trailing `?`.

* Only show live proofs.

* change valid to active in proof list and update liveness before displaying

* minor fixes

* add keybase config at well-known path

* extremely naive feature flagging off the identity proof UI

* fixes for rubocop

* make identity proofs page resilient to potential keybase issues

* normalize i18n

* tweaks for brakeman

* remove two unused translations

* cleanup and add more localizations

* make keybase_contacts an admin setting

* fix ExternalProofService my_domain

* use Addressable::URI in identity proofs

* use active model serializer for keybase proof config

* more cleanup of keybase proof config

* rename proof is_valid and is_live to proof_valid and proof_live

* cleanup

* assorted tweaks for more robust communication with keybase

* Clean up

* Small fixes

* Display verified identity identically to verified links

* Clean up unused CSS

* Add caching for Keybase avatar URLs

* Remove keybase_contacts setting
2019-03-18 21:00:55 +01:00
ThibG a20354a20b Set and store report URIs (#10303)
Fixes #10271
2019-03-17 15:34:56 +01:00
ThibG 5e38ef87a7 Fix reblogs privacy (#10302)
* Fix reblogs privacy

* Fix Announce processing specs
2019-03-17 14:54:09 +01:00
Eugen Rochko 1c113fd72d
Add relationship manager UI (#10268) 2019-03-16 11:23:22 +01:00
Eugen Rochko 3ad3223b46
Fix detailed poll validation errors not being returned in the API (#10261)
No more "Owned poll is invalid"
2019-03-15 13:36:38 +01:00
Eugen Rochko ba84b6d4d7
Add `visibility` param to reblog REST API (#9851)
Use async worker for creating reblog notification to improve performance
2019-03-15 04:36:41 +01:00
Eugen Rochko 51e154f5e8
Admission-based registrations mode (#10250)
Fix #6856
Fix #6951
2019-03-14 05:28:30 +01:00
Eugen Rochko 9e33174604
Refactor User model, extract PamAuthenticable, LdapAuthenticable (#10217) 2019-03-14 02:13:42 +01:00
ThibG 65d9004ac9 Add UI for enabling/disabling poll notifications (#10255)
* Add UI for enabling/disabling poll notifications

* Add poll notifications to the (advanced) quick filter bar

* Update poll notification message

“Your poll has ended” → “A poll you have voted in has ended”

* Clear up associated notifications when a poll is deleted
2019-03-13 19:29:54 +01:00
Eugen Rochko 06663fcf87
Fix `tagged` param not being normalized before querying tags (#10249) 2019-03-13 13:02:13 +01:00
Eugen Rochko 9f5b55ad4f
Fix poll update handler calling method was that was not available (#10246)
* Fix poll update handler calling method was that was not available

Fix regression from #10209

* Refactor VoteService

* Refactor ActivityPub::DistributePollUpdateWorker and optimize it

* Fix typo

* Fix typo
2019-03-12 22:58:59 +01:00
Aurélien Reeves 85537b0069 Squish username before validation (#10239)
* Squish username before validation (#10101)

Fix #10101

* Move before_validation hook to a private method

Also add Unicode wite-spaces to the spec to support the use of squish
over strip.
2019-03-11 20:48:24 +01:00
ThibG 3a92885a86 Support pushing and receiving updates to poll tallies (#10209)
* Process incoming poll tallies update

* Send Update on poll vote

* Do not send Updates for a poll more often than once every 3 minutes

* Include voters in people to notify of results update

* Schedule closing poll worker on poll creation

* Add new notification type for ending polls

* Add front-end support for ended poll notifications

* Fix UpdatePollSerializer

* Fix Updates not being triggered by local votes

* Fix tests failure

* Fix web push notifications for closing polls

* Minor cleanup

* Notify voters of both remote and local polls when those close

* Fix delivery of poll updates to mentioned accounts and voters
2019-03-11 00:49:31 +01:00
ThibG 3f4a6d44fe Fix setting up fields of a previously suspended account (#10222)
Fix #10177 for real
2019-03-08 19:59:08 +01:00
ThibG 3aaac4f134 Do not allow adding votes to expired polls (#10214)
* Do not allow adding votes to expired polls

* Only validate expires_at on create
2019-03-08 00:54:50 +01:00
Eugen Rochko 054bbb3da2
Immediately display poll results to poll author (#10187)
* Immediately display poll results to poll author

* Refactor Poll#loaded_options and add Poll#voted? to improve DRYness
2019-03-07 22:53:47 +01:00
ThibG 96f905f409 Add optimistic lock to avoid race conditions when handling votes (#10196)
* Add optimistic lock to avoid race conditions when handling votes

* Force-reload polls when getting `ActiveRecord::StaleObjectError`
2019-03-06 19:53:57 +01:00
Eugen Rochko 0c43c320db
Fix status creation API silently discarding invalid poll (#10171) 2019-03-05 15:21:31 +01:00
ThibG d785497ba5 Fix suspended account's fields being set as empty dict instead of list (#10178)
Fixes #10177
2019-03-05 15:19:54 +01:00
Eugen Rochko a198add83b
Fix various issues in polls (#10165)
* Fix ActivityPub poll results being serialized even with hide_totals

* Fix poll refresh button having a different font size

* Display poll in OpenGraph description

* Fix NoMethodError when serializing votes

Regression from #10158

* Fix polls on public pages being broken for non-logged-in users

* Do not show time remaining if poll has no expiration date
2019-03-05 03:51:18 +01:00
Eugen Rochko 5d3e7cee99
Fix featured tag form not failing on failed tag validations (#10167) 2019-03-05 03:46:24 +01:00
Eugen Rochko 05dfd632c7
Fix poll options not being stripped of surrounding whitespace on save (#10168) 2019-03-05 03:45:56 +01:00
ThibG 833ffce2df Store remote votes URI (#10158)
* Store remote votes URI

* Add spec for accepting remote votes

* Make poll vote id generation work the same way as follows
2019-03-04 22:51:23 +01:00
ThibG e13d3792f3 Make sure the poll is created before storing its id (#10142)
* Make sure the poll is created before storing its id

* Fix updating poll results

* Support fetching Question activities from the search bar
2019-03-04 00:39:06 +01:00
Eugen Rochko 230a012f00
Add polls (#10111)
* Add polls

Fix #1629

* Add tests

* Fixes

* Change API for creating polls

* Use name instead of content for votes

* Remove poll validation for remote polls

* Add polls to public pages

* When updating the poll, update options just in case they were changed

* Fix public pages showing both poll and other media
2019-03-03 22:18:23 +01:00
ThibG 99dc212ae5 Fix lists export (#10136) 2019-03-03 15:38:47 +01:00
ThibG 9d3c6f1849 Improved remote thread fetching (#10106)
* Fetch up to 5 replies when discovering a new remote status

This is used for resolving threads downwards. The originating
server must add a “replies” attributes with such replies for it to
be useful.

* Add some tests for ActivityPub::FetchRepliesWorker

* Add specs for ActivityPub::FetchRepliesService

* Serialize up to 5 public self-replies for ActivityPub notes

* Add specs for ActivityPub::NoteSerializer

* Move exponential backoff logic to a worker concern

* Fetch first page of paginated collections when fetching thread replies

* Add specs for paginated collections in replies

* Move Note replies serialization to a first CollectionPage

The collection isn't actually paginable yet as it has no id nor
a `next` field. This may come in another PR.

* Use pluck(:uri) instead of map(&:uri) to improve performances

* Fix fetching replies when they are in a CollectionPage
2019-02-28 15:22:21 +01:00
Eugen Rochko e7f20cc43f
Add type, limit, offset, min_id, max_id, account_id to search API (#10091)
* Add type, limit, offset, min_id, max_id, account_id to search API

Fix #8939

* Make the offset work on accounts and hashtags search as well

* Assure brakeman we are not doing mass assignment here

* Do not allow paginating unless a type is chosen

* Fix search query and index id field on statuses instead of created_at
2019-02-26 15:21:36 +01:00
Hinaloe 4340d338fd correct opt-out showing application (#10086)
* correct opt-out showing application

refs #9994

* Revert "correct opt-out showing application"

This reverts commit 0e9bb70f145be42962416a6b87c08d59a2896486.

* User#shows_application? calls wrong value
2019-02-20 17:47:38 +01:00
ThibG 8e7fc7ec73 Fix crash when conversations have no valid participants (#10078)
* Never return empty participants for conversations

Fixes #10068

* Fix client-side crash when conversations have no participants
2019-02-19 20:00:41 +01:00
ThibG 6840a77711 Add domain search/filter to the "Federation" (/admin/instances) page (#10071) 2019-02-18 14:59:19 +01:00
Eugen Rochko 8ef50706a1
Fix relay enabling/disabling not resetting inbox availability status (#10048)
Fix #10033
2019-02-15 16:08:59 +01:00
rinsuki c78d64d9c3 Use video filesize limit with gifv (#9924) 2019-02-04 04:46:05 +01:00
Eugen Rochko 364f2ff9aa
Add featured hashtags to profiles (#9755)
* Add hashtag filter to profiles

GET /@:username/tagged/:hashtag
GET /api/v1/accounts/:id/statuses?tagged=:hashtag

* Display featured hashtags on public profile

* Use separate model for featured tags

* Update featured hashtag counters on-write

* Limit featured tags to 10
2019-02-04 04:25:59 +01:00
Eugen Rochko d14c276e58
Add option to overwrite imported data (#9962)
* Add option to overwrite imported data

Fix #7465

* Add import for domain blocks
2019-02-03 03:59:51 +01:00
ThibG ed30110618 Make displaying application used to toot opt-in (#9897)
* Make storing and displaying application used to toot opt-in

* Revert to storing application info, and display it to the author via API
2019-02-02 19:18:15 +01:00
ysksn bcfff65195 Create Redisable#redis (#9633)
* Create Redisable

* Use #redis instead of Redis.current
2019-02-02 19:11:38 +01:00
Acid Chicken (硫酸鶏) 5092d17f29 Add WebP support (#9879)
* Add WebP support

* Remove the changes to the tooltip

refs: https://github.com/tootsuite/mastodon/pull/9879#pullrequestreview-199312528
2019-02-02 12:25:04 +01:00
Eugen Rochko 9519d55332
Fix SSO authentication not working due to missing agreement boolean (#9915)
Fix #9906
2019-01-25 12:36:54 +01:00
Eugen Rochko 80768e2840
Fix missing account association in tombstone model (#9857) 2019-01-18 23:52:09 +01:00
ThibG 75b1488cf4 Add tombstones for remote statuses (#9830)
* Add Tombstone model to remember object deletion

* Do not recreate a status if it has been deleted

* Record Tombstone for remote deleted items

Also, only record deleted items from same-host actors

* Clear an user's tombstones when their key change
2019-01-18 15:56:55 +01:00
Eugen Rochko 31f396b57d
Add support for non-public reblogs from ActivityPub (#9841)
Fix #9838
2019-01-18 15:56:21 +01:00
Eugen Rochko 69f782b54d
Fix code style of regeneration-related code (#9843) 2019-01-18 01:02:51 +01:00
Eugen Rochko 1c6588accc
Redesign admin instances area (#9645) 2019-01-08 13:39:49 +01:00
Eugen Rochko 86f27b178c
Fix deleting a scheduled status immediately deleting media attachments (#9728) 2019-01-06 16:38:40 +01:00
Eugen Rochko a49d43d112
Add scheduled statuses (#9706)
Fix #340
2019-01-05 12:43:28 +01:00
Naoki Kosaka fae32634b1 Use Contact User as Relay, Report, Subscribe. (#9661)
* Use Contact User as Relay, Report, Subscribe.

* Use Account.representative to fetch contact user.

* Use find_local.

* No reason to use Account.representative in subscribe_service.

* Don't required representative!

* Fallback is included in Account.representative method.
2019-01-05 07:17:12 +01:00
Eugen Rochko 45899cfa69
Change note length validation to ignore mention domains and URLs (#9717)
Fix #4419
2019-01-05 07:16:46 +01:00
Eugen Rochko 66436d0895
Improve e-mail digest (#9689)
- Reduce time-to-digest from 20 to 7 days
- Fetch mentions starting from +1 day since last login
- Fix case when last login is more recent than last e-mail
- Do not render all mentions, only 40, but show number in subject
- Do not send digest to moved accounts
- Do send digest to silenced accounts
2019-01-02 10:47:32 +01:00
Eugen Rochko 7fc7437d05
Add CSV export for lists and domain blocks (#9677)
Fix #6893
Fix #9268
2019-01-01 13:44:04 +01:00
Eugen Rochko 0f938ff29c
Add handler for Move activity (#9629) 2018-12-29 02:24:36 +01:00
Eugen Rochko 6e49907ecf
Improve admin UI for account view (#9643) 2018-12-28 03:38:41 +01:00
chr v1.x c3465f699e Add local followers page to admin account UI (#9610)
* Add local followers page to admin account UI

For moderation, I often find myself wondering who, locally, is following
a remote user. Currently, to see this, I have to go back to the web UI,
paste in their full handle, click their profile, and go to the
"Followers" tab (plus, this information is incidental, and if mastodon
ever decides to resolve all of the follower information, there will be
no place local followers are shown). This PR adds a new page which is
accessible via the "following" count on the admin's account view
page, which shows the local followers. (It has filter parameters for
account location to indicate that only local followers are shown, and
leave room for expansion if mastodon ever decides to store the entire
remote follow list).

* Normalize en.yml
2018-12-27 13:15:39 +01:00
ysksn a8ba291beb Fix scope latest of ReportNote (#9630) 2018-12-26 06:38:59 +01:00
ysksn 0c1e4bb969 Create DomainNormalizable#normalize_domain (#9631) 2018-12-26 06:38:42 +01:00
ysksn 6641a1cac9 Remove Form::StatusBatch::ACTION_TYPE unused (#9623) 2018-12-25 18:29:51 +01:00
Eugen Rochko 5d2fc6de32
Add REST API for creating an account (#9572)
* Add REST API for creating an account

The method is available to apps with a token obtained via the client
credentials grant. It creates a user and account records, as well as
an access token for the app that initiated the request. The user is
unconfirmed, and an e-mail is sent as usual.

The method returns the access token, which the app should save for
later. The REST API is not available to users with unconfirmed
accounts, so the app must be smart to wait for the user to click a
link in their e-mail inbox.

The method is rate-limited by IP to 5 requests per 30 minutes.

* Redirect users back to app from confirmation if they were created with an app

* Add tests

* Return 403 on the method if registrations are not open

* Require agreement param to be true in the API when creating an account
2018-12-24 19:12:38 +01:00
Eugen Rochko 3c033c4352
Add moderation warnings (#9519)
* Add moderation warnings

Replace individual routes for disabling, silencing, and suspending
a user, as well as the report update route, with a unified account
action controller that allows you to select an action (none,
disable, silence, suspend) as well as whether it should generate an
e-mail notification with optional custom text. That notification,
with the optional custom text, is saved as a warning.

Additionally, there are warning presets you can configure to save
time when performing the above.

* Use Account#local_username_and_domain
2018-12-22 20:02:09 +01:00
Eugen Rochko adaf249700
Fix regression in #9539 (#9541) 2018-12-17 04:32:36 +01:00
Eugen Rochko 087e118971
Remove "most popular" tab from profile directory, add responsive design (#9539)
* Remove "most popular" tab from profile directory, add responsive design

* Remove unused translations
2018-12-17 03:14:28 +01:00
Eugen Rochko 628da11e38
Do no retry web push workers if the server returns a 4xx response (#9434)
Add timeout of 10s to web push requests
2018-12-17 03:14:13 +01:00
ThibG 720daa8143 Add instance-wide setting to disable profile directory (#9497)
* Add instance-wide setting to disable profile directory

Fixes #9496

When the profile directory is disabled:
- The “discoverable” setting is hidden from users
- The “profile directory” link is not shown on public pages
- /explore returns 404

* Move Setting.profile_directory check to a before_action filter
2018-12-11 19:18:29 +01:00
Adam Copp 7d00e4edbd Make custom emoji domains case insensitive #9351 (#9474)
* Make custom emoji domains case sensitive #9351

* Fixup style in downcase_domain to comply with codeclimate.

* switch if! to unless

* Don't use transactions, operate in batches.

Also revert spurious schema change.
2018-12-11 05:30:57 +01:00
Eugen Rochko dbb1ee269f
Improve e-mail MX validator and add tests (#9489) 2018-12-10 22:53:25 +01:00
ThibG 81bda7d67c Add setting to not aggregate reblogs (#9248)
* Add setting to not aggregate reblogs

Fixes #9222

* Handle cases where user is nil in add_to_home and add_to_list

* Add hint for setting_aggregate_reblogs option

* Reword setting_aggregate_reblogs label
2018-12-09 13:03:01 +01:00
Eugen Rochko 6b78e5b5ab
Cache hashtag sample accounts, and exclude ineligible ones (#9465) 2018-12-08 01:32:26 +01:00
Eugen Rochko ecd303c097
Fix various things in the directory (#9449)
* Fix missing variable in directory page title

* Order hashtags by number of people instead of alphabetically

* Add icon to OpenGraph preview of directory page

* Prevent line breaks in hashtags and ensure lowercase in the table
2018-12-07 16:37:32 +01:00
Takeshi Umeda c1c0f7c516 Fix tag mb_chars comparison of profile directory (#9448) 2018-12-07 04:18:37 +01:00
Eugen Rochko 73be8f38c1
Add profile directory (#9427)
Fix #5578
2018-12-06 17:36:11 +01:00
ThibG 1a22eff1e0 Attempt fixing deadlocks by moving account stats update outside transaction (#9437)
* Use `update_column` instead of `update_attribute` in callback

`update_attribute` would normally cause callbacks to be called.
Called from a callback, it seems to stop further callbacks from executing.

`update_column` does the same work, but without calling callbacks or
preventing other callbacks from executing.

* Fix deadlocks by moving account stats update outside transaction
2018-12-05 22:51:12 +01:00
Eugen Rochko 6ddf0432e7
Improve account suspension speed and completeness (#9290)
- Some associations were missing from the clean-up
- Some attributes were not reset on suspension
- Skip federation and streaming deletes when purging a dead domain
- Move account association definitions to concern
2018-12-03 01:32:08 +01:00
Eugen Rochko 4afda5444a
Change identities id column to a bigint (#9371)
* fix: change Identity's id column to a bigint

This appears to be the last model created using a 5.0 migration,
where column types defaulted to `integer` rather than `bigint`.

This migration changes the column type to match that of all of the
other ID columns.

* Change user_id column in identities to bigint and fix down-migration
2018-11-27 13:56:25 +01:00
ThibG 395615d9f3 Allow hyphens in the middle of remote user names (#9345)
Fixes #9309

This only allows hyphens in the middle of a username, much like dots,
although I don't have a compelling reason to do so other than keeping
the changes minimal.
2018-11-27 12:28:01 +01:00
Hugo Gameiro b6681d21ad add loglevel to ffmpeg in gif upload (#9368) 2018-11-27 12:19:12 +01:00
Eugen Rochko 73faadad28
Redesign admin accounts index (#9340)
* Improve overview of accounts in admin UI

- Display suspended status, role, last activity and IP prominently
- Default to showing local accounts
- Default to not showing suspended accounts

* Remove unused strings

* Fix tests

* Allow filtering accounts by IP mask
2018-11-26 15:53:27 +01:00
valerauko db9aea34de Ensure replied-to is a status not a boost (#9129)
* Ensure replied-to is a status not a boost

* Consider case of not a reply

* Add test case for replying to boost

* Move reblog-reply resolution to model

* Remove unnecessary comment
2018-11-25 16:35:21 +01:00
Eugen Rochko 0eaf6d7693
Sort self-replies to the top of descendants (#9320)
Fix #6463
2018-11-24 20:48:50 +01:00
Eugen Rochko 87a43274f1
Revert "Sort self-replies to the top of the thread (#9296)" (#9335)
This reverts commit 074960bb0f.

Fix #9315
2018-11-23 21:58:01 +01:00
Eugen Rochko d6b9a62e0a
Extract counters from accounts table to account_stats table (#9295) 2018-11-19 00:43:52 +01:00
Eugen Rochko 074960bb0f
Sort self-replies to the top of the thread (#9296)
Fix #6463
2018-11-16 19:34:10 +01:00
Eugen Rochko 8069fd636b
Remove intermediary arrays when creating hash maps from results (#9291) 2018-11-16 15:02:18 +01:00
James Kiesel 4c03e05a4e Allow joining several hashtags in a single column (#8904)
* Nascent tag menu on frontend

* Hook up frontend to search

* Tag intersection backend first pass

* Update yarnlock

* WIP

* Fix for tags not searching correctly

* Make radio buttons function

* Simplify radio buttons with modeOption

* Better naming

* Rearrange options

* Add all/any/none functionality on backend

* Small PR cleanup

* Move to service from scope

* Small cleanup, add proper service tests

* Don't use send with user input :D

* Set appropriate column header

* Handle auto updating timeline

* Fix up toggle function

* Use tag value correctly

* A bit more correct to use 'self' rather than 'all' in status scope

* Fix some style issues

* Fix more code style issues

* Style select dropdown more better

* Only use to_id'ed value to ensure no SQL injection

* Revamp frontend to allow for multiple selects

* Update backend / col header to account for more flexible tagging

* Update brakeman ignore

* Codeclimate suggestions

* Fix presenter tag_url

* Implement initial PR feedback

* Handle additional tag streaming

* CodeClimate tweak
2018-11-05 18:53:25 +01:00
Eugen Rochko 11b3ee4f4c
Reset status cache when status_stat or media_attachment updates (#9119)
* Reset status cache when status_stat or media_attachment updates

Fix #8711

Media attachments are generally immutable, but admins can update
the sensitive flag, and this would ensure the change is visible
instantly. Same for updates to status stats. That is a regression
from #8185, because even the correct updated_at fetched from a join
doesn't seem to invalidate the cache.

* Remove join from Status#cache_ids since it has no effect
2018-10-28 06:42:34 +01:00
Eugen Rochko 795f0107d2
Include preview cards in status entity in REST API (#9120)
* Include preview cards in status entity in REST API

* Display preview card in-stream

* Improve in-stream display of preview cards
2018-10-28 06:35:03 +01:00
Eugen Rochko 4ea718ef18
Migrate all old direct messages to new conversations schema (#9085) 2018-10-25 01:44:02 +02:00
Eugen Rochko fd5285658f
Add option to block reports from domain (#8830) 2018-10-20 08:02:44 +02:00
Eugen Rochko a38a452481
Add unread indicator to conversations (#9009) 2018-10-19 01:47:29 +02:00
Eugen Rochko 72d7d3003b
Do not show "limited" visibility in default visibility preference (#8999)
* Do not show "limited" visibility in default visibility preference

Fix regression from #8950

* Fix code style issue
2018-10-17 22:04:40 +02:00
Eugen Rochko ddd30f331c
Improve support for aspects/circles (#8950)
* Add silent column to mentions

* Save silent mentions in ActivityPub Create handler and optimize it

Move networking calls out of the database transaction

* Add "limited" visibility level masked as "private" in the API

Unlike DMs, limited statuses are pushed into home feeds. The access
control rules between direct and limited statuses is almost the same,
except for counter and conversation logic

* Ensure silent column is non-null, add spec

* Ensure filters don't check silent mentions for blocks/mutes

As those are "this person is also allowed to see" rather than "this
person is involved", therefore does not warrant filtering

* Clean up code

* Use Status#active_mentions to limit returned mentions

* Fix code style issues

* Use Status#active_mentions in Notification

And remove stream_entry eager-loading from Notification
2018-10-17 17:13:04 +02:00
ashleyhull-versent f194857ac9 rubocop issues - Cleaning up (#8912)
* cleanup pass

* undo mistakes

* fixed.

* revert
2018-10-08 04:50:11 +02:00
Eugen Rochko 774ac47373
Add conversations API (#8832)
* Add conversations API

* Add web UI for conversations

* Add test for conversations API

* Add tests for ConversationAccount

* Improve web UI

* Rename ConversationAccount to AccountConversation

* Remove conversations on block and mute

* Change last_status_id to be a denormalization of status_ids

* Add optimistic locking
2018-10-07 23:44:58 +02:00
Eugen Rochko a46ab86adf
Limit the number of people that can be followed from one account (#8807)
Configurable soft limit of 7,500, and above that, configurable
ratio of 1.1 * followers, controlled by:

- MAX_FOLLOWS_THRESHOLD
- MAX_FOLLOWS_RATIO

Fix #2311
2018-10-04 17:36:11 +02:00
Eugen Rochko e645ae9561
Change admin accounts default sort to most recent (#8813) 2018-10-04 16:05:38 +02:00
Eugen Rochko 7fe137d2f7
Fix link verification for remote accounts (#8868) 2018-10-04 15:47:03 +02:00
Eugen Rochko 6c835085a3
Fix timeline pagination (#8827)
Ruby's ** operator does not play well with non-Hash objects, which
the params slice is

Fix #8821
2018-09-29 19:03:33 +02:00
Eugen Rochko f0fff3eb10
Support min_id-based pagination in REST API (#8736)
* Allow min_id pagination in Feed#get

* Add min_id pagination to home and list timeline APIs

* Add min_id pagination to account statuses, public and tag APIs

* Remove unused stub in reports API

* Use min_id pagination in notifications, favourites, and fix order

* Fix HomeFeed#from_database not using paginate_by_id
2018-09-28 02:23:45 +02:00
ふぁぼ原 f7a6f9489d Add a new preference to always hide all media (#8569) 2018-09-25 05:09:35 +02:00
Matt Sweetman 1889526e23 Add user preference to always expand toots marked with content warnings (#8762) 2018-09-24 05:44:01 +02:00
Eugen Rochko bac8227519
Fix performance regression in Account::Field#verifiable? (#8719)
* Fix performance regression in Account::Field#verifiable?

Regression from #8703

* Fix code style issue
2018-09-18 22:49:24 +02:00
Eugen Rochko f4d549d300
Redesign forms, verify link ownership with rel="me" (#8703)
* Verify link ownership with rel="me"

* Add explanation about verification to UI

* Perform link verifications

* Add click-to-copy widget for verification HTML

* Redesign edit profile page

* Redesign forms

* Improve responsive design of settings pages

* Restore landing page sign-up form

* Fix typo

* Support <link> tags, add spec

* Fix links not being verified on first discovery and passive updates
2018-09-18 16:45:58 +02:00
luzpaz 40dd19be37 Misc. typos (#8694)
Found via `codespell -q 3 --skip="./app/javascript/mastodon/locales,./config/locales"`
2018-09-14 00:53:09 +02:00
Eugen Rochko c593d6df9c
Add preference for report notification e-mails, skip for duplicates (#8559)
If an unresolved report for the same target account already exists,
no new notification is generated
2018-09-02 00:11:58 +02:00
Renato "Lond" Cerqueira 5b2b493a90 Fix nil host in remotable (#8508)
Host can be nil in urls like
'https:https://example.com/path/file.png'
2018-08-29 21:13:49 +02:00
Quint Guvernator da13fa5021 Fix low-hanging rubocop gripes (#8458)
* rubocop: quit being so picky

* rubocop: miscellany

* rubocop: prefer present to blank
2018-08-26 19:22:46 +02:00
Eugen Rochko a2cabf3f4a
Add admin custom CSS setting (#8399)
Fix #3894
2018-08-24 04:33:27 +02:00
Eugen Rochko 2f34b747b3
Allow mods to disable login, improve message when login disabled (#8329)
* Allow moderators to disable/enable login

* Instead of rejecting login, show forbidden error when login disabled

Avoid confusion because when login is rejected, the message is that
the account is not activated, which is wrong.

* Fix tests
2018-08-23 23:26:29 +02:00
Jakub Mendyk 6cb3514d64 Add ability to change an instance default theme from the administration panel (#7092) (#8381)
* Add default_settings class method to ScopedSettings

ScopedSettings was extended to use value of unscoped setting instead of
only using defaults set in config/settings.yml for selected settings.
This adds possibility for admins to set default values of users' settings,
for example default theme (as requested in #7092).

* Add ability to change an instance default theme

Closes #7092
2018-08-23 14:17:35 +02:00
masarakki 4bdab203ac exclude-other-silenced-accounts (#7528) 2018-08-22 13:20:50 +02:00
Eugen Rochko 2374a00c10
Add confirmation step to account suspensions (#8353)
* Add confirmation page for suspensions

* Suspension confirmation closes reports, linked from report UI

* Fix tests
2018-08-22 11:53:41 +02:00
Eugen Rochko d98de8ada7
Get rid of all batch order warnings (#8334) 2018-08-21 12:25:50 +02:00
Eugen Rochko 83746b6364
Remove annoying Notification#cache_ids scope warning (#8333) 2018-08-21 02:49:51 +02:00
ThibG b34d6238cb Add API endpoint to list featured accounts (fixes #8315) (#8317) 2018-08-20 18:46:04 +02:00
abcang 9e75aa30cd Unuse ActiveRecord::Base#cache_key (#8185)
* Unuse ActiveRecord::Base#cache_key

* Enable cache_versioning

* Call cache_collection
2018-08-19 15:52:38 +02:00
Eugen Rochko 59c68c1a74
Reduce user active duration from 7 days to 2 days (#8282)
To minimize fanout work and redis home feed storage space when
there are lots of recent sign-ups
2018-08-19 03:50:34 +02:00
Eugen Rochko d010816ba8
Fix error when trying to update counters for statuses that are gone (#8251) 2018-08-18 03:03:23 +02:00
Eugen Rochko 78fa926ed5
Add remote interaction dialog for toots (#8202)
* Add remote interaction dialog for toots

* Change AuthorizeFollow into AuthorizeInteraction, support statuses

* Update brakeman.ignore

* Adjust how interaction buttons are display on public pages

* Fix tests
2018-08-18 03:03:12 +02:00
ThibG 59f7f4c923 Implement Undo { Accept { Follow } } (fixes #8234) (#8245)
* Add Follow#revoke_request!

* Implement Undo { Accept { Follow } } (fixes #8234)
2018-08-17 16:24:56 +02:00
Eugen Rochko d78474264d
Update reply counters only if the reply is public/unlisted (#8211) 2018-08-16 14:21:52 +02:00
Eugen Rochko aaac14b8ad
Show exact number of followers/statuses on export page/in tooltip (#8199)
* Show exact number of followers/statuses on export page/in tooltip

* Fix tests
2018-08-14 21:56:17 +02:00
Eugen Rochko 8e111b753a
Move status counters to separate table, count replies (#8104)
* Move status counters to separate table, count replies

* Migration to remove old counter columns from statuses table

* Fix schema file
2018-08-14 19:19:32 +02:00
Eugen Rochko 39e361a56d
Expect relays to answer with accept/reject (#8179) 2018-08-12 18:16:26 +02:00
Eugen Rochko aff6a15061
When follow is removed, remove endorsement (#8149) 2018-08-09 15:24:29 +02:00
Eugen Rochko f2404de871
Public profile endorsements (accounts picked by profile owner) (#8146) 2018-08-09 09:56:53 +02:00
Eugen Rochko cc56f2230a
Add separate setting for sidebar text (site_short_description) (#8107)
* Add separate setting for sidebar text (site_short_description)

* Fix tests
2018-07-31 18:59:34 +02:00
Eugen Rochko e7e577dd6e
Enforce username format for remote users, too (#8102)
Initially I thought there might be valid reasons for remote users to
have a different, unpredicted username format. However, I now realize
such a difference would be unusable and unexpected within Mastodon.

Fix #8058
2018-07-30 22:29:52 +02:00
Eugen Rochko 1656663598
Convert MOV and WEBM to MP4, raise maximum limit to 40MB (#8101)
Separate size limits for images and video. Images remain at 8MB,
while videos can be up to 40MB.
2018-07-30 19:33:05 +02:00
Eugen Rochko bb71538bb5
Redesign public profiles and toots (#8068) 2018-07-28 19:25:33 +02:00
Eugen Rochko 0fb0037ca7
Resize images by area instead of fixed dimensions (#8083)
To improve the way super tall or super ride images are treated, the
numbers remain the same, 1280x1280 and 400x400, but if an image
is less in one dimension than the other, the other can become larger

Thanks to @WAHa_06x36@mastodon.social for the tip
2018-07-28 03:33:00 +02:00
Eugen Rochko 69c34f3438
Fix number of results returned from TrendingTags.get 2018-07-16 19:54:14 +02:00
Eugen Rochko 0d13e30ad2
Display trending hashtags on admin dashboard (#8038) 2018-07-16 19:21:27 +02:00
Eugen Rochko e55dce3176
Add federation relay support (#7998)
* Add federation relay support

* Add admin UI for managing relays

* Include actor on relay-related activities

* Fix i18n
2018-07-13 02:16:06 +02:00
ThibG 1ca4e51eb3 Add option to not consider word boundaries when processing keyword filtering (#7975)
* Add option to not consider word boundaries when filtering phrases

* Add a few tests for keyword/phrase filtering
2018-07-09 02:22:09 +02:00
Eugen Rochko 6b9e03e002
Add API method to remove a suggestion (#7978)
DELETE /api/v1/suggestions/:account_id

When blocking, remove suggestion from both sides. Muting not affected,
since muting is supposed to be invisible to the target.
2018-07-07 21:09:54 +02:00
Eugen Rochko f89c595ea0
Add admin setting to enable OG previews for sensitive media (#7962) 2018-07-06 02:15:56 +02:00
Renato "Lond" Cerqueira 4045b50bd6 Restore support to ruby 2.3, add ruby 2.3 to circle ci (#7935)
This replace calls of String#match? with rails Regex#match?
This follows the same idea used to keep Rails 5.2 compatible with Ruby
2.2.2 in https://github.com/rails/rails/pull/32973
2018-07-03 19:47:09 +02:00
Eugen Rochko da8fe8079e
Re-add follow recommendations API (#7918)
* Re-add follow recommendations API

    GET /api/v1/suggestions

Removed in 8efa081f21 due to Neo4J
dependency. The algorithm uses triadic closures, takes into account
suspensions, blocks, mutes, domain blocks, excludes locked and moved
accounts, and prefers more recently updated accounts.

* Track interactions with people you don't follow

Replying to, favouriting and reblogging someone you're not following
will make them show up in follow recommendations. The interactions
have different weights:

- Replying is 1
- Favouriting is 10 (decidedly positive interaction, but private)
- Reblogging is 20

Following them, muting or blocking will remove them from the list,
obviously.

* Remove triadic closures, ensure potential friendships are trimmed
2018-07-03 01:47:56 +02:00
Eugen Rochko cdb101340a
Keyword/phrase filtering (#7905)
* Add keyword filtering

    GET|POST       /api/v1/filters
    GET|PUT|DELETE /api/v1/filters/:id

- Irreversible filters can drop toots from home or notifications
- Other filters can hide toots through the client app
- Filters use a phrase valid in particular contexts, expiration

* Make sure expired filters don't get applied client-side

* Add missing API methods

* Remove "regex filter" from column settings

* Add tests

* Add test for FeedManager

* Add CustomFilter test

* Add UI for managing filters

* Add streaming API event to allow syncing filters

* Fix tests
2018-06-29 15:34:36 +02:00
ThibG 50a2854f92 Fix jpeg files sometimes being returned with a .jpe extension (#7881)
While this isn't exactly *wrong*, files uploaded with a “.jpe” extension will
keep that extension, which will often cause them to be served with an
incorrect mimetype.
2018-06-24 13:33:06 +02:00
Eugen Rochko 50689f0d41
Create special case to prefer "jpeg" over "jpe" file extension (#7841) 2018-06-18 17:27:05 +02:00
Eugen Rochko a58ec29631
Allow selecting default posting language instead of auto-detect (#7828)
* Allow selecting default posting language instead of auto-detect

* Enable default language setting in credentials API

* Fix form saving
2018-06-17 18:57:31 +02:00
Eugen Rochko 7eec279c7f
Change language opt-out to language opt-in (#7823)
* Switch filtered_languages to chosen_languages

* Adjust interface

* Remove unused translations
2018-06-17 13:54:02 +02:00
Eugen Rochko ca85658975
Add autofollow option to invites (#7805)
* Add autofollow option to invites

* Trigger CodeClimate rebuild
2018-06-15 18:00:23 +02:00
Eugen Rochko 17dc07f667
Fix has_one association on Web::PushSubscription (#7767)
* Fix has_one association on Web::PushSubscription

* Fix code style issues
2018-06-09 21:11:14 +02:00
Eugen Rochko 59b42188a7
Filter out blocked/muted people from profile timelines (#7747)
Fix #7741
2018-06-06 21:13:30 +02:00
Yamagishi Kazutoshi 683707839f Fix fetch of remote image with multiple Content-Type headers (#7749) 2018-06-06 20:50:07 +02:00
Yamagishi Kazutoshi b7b331ad0d Add missing dot for remote image (#7751) 2018-06-06 20:49:39 +02:00
Eugen Rochko bd0791d800
Add redraft function (#7735)
* Add redraft function

Fix #7010

* Add explicit confirmation

* Add explicit confirmation message
2018-06-05 00:17:38 +02:00
nightpool 5fb013878f Fix context performance by partially reverting #7083 (#7734)
* Fix context performance by partially reverting #7083

* Fix code style issue

* fix off-by-1 error in thread limits

* code style fix
2018-06-04 22:20:12 +02:00
Yamagishi Kazutoshi dc73241bd9 Detect extname from Content-Type (#7733) 2018-06-04 18:58:36 +02:00
Akihiko Odaki 5bf5003384 Do not mark remote status sensitive even if spoiler text is present (#7395)
Old statuses and statuses from Pawoo, which runs a modified version of
Mastodon, may not have been marked sensitive even if spoiler text is
present.

Such statuses are still not marked sensitve if they are local or
arrived before version upgrade. Marking recently fetched remote status
sensitive contradicts the behavior.

Considering what people expected when they authored such statuses, this
change removes the sensitivity enforcement.
2018-06-04 04:46:14 +02:00
Eugen Rochko 0deb9fa6b9
Remove trending hashtags (#7711)
* Delete trends_controller.rb

* Update routes.rb

* Update trending_tags.rb

* Update index.js

* Update index.js

* Update search_results.js

* Update async-components.js

* Update index.js

* Delete trends.js

* Delete trends.js

* Delete trends_container.js

* Delete trends.js

* Update search_results.js

* Update search_results_container.js
2018-06-04 02:18:18 +02:00
Eugen Rochko 6e6c0e9613
Use different algorithm for trending tags (#7697) 2018-06-01 19:19:44 +02:00
Eugen Rochko a7d726c383
Improve counter caches on Status and Account (#7644)
Do not touch statuses_count on accounts table when mass-destroying
statuses to reduce load when removing accounts, same for
reblogs_count and favourites_count

Do not count statuses with direct visibility in statuses_count

Fix #828
2018-05-30 02:50:23 +02:00
Eugen Rochko 461542784b
Reduce wasted work in RemoveStatusService due to inactive followers (#7672) 2018-05-29 22:55:33 +02:00
Eugen Rochko e599d7caf2 Rescue Mastodon::DimensionsValidationError in Remoteable (#7662)
Fix #7660
2018-05-29 08:39:02 +09:00
tateisu b87a1229c7 optimize direct timeline (#7614)
* optimize direct timeline

* fix typo in class name

* change filter condition for direct timeline

* fix codestyle issue

* revoke index_accounts_not_silenced because direct timeline does not use it.

* revoke index_accounts_not_silenced because direct timeline does not use it.

* fix rspec test condition.

* fix rspec test condition.

* fix rspec test condition.

* revoke adding column and partial index

* (direct timeline) move merging logic to model

* fix pagination parameter

* add method arguments that switches return array of status or cache_ids

* fix order by

* returns ActiveRecord.Relation in default behavor

* fix codestyle issue
2018-05-28 11:04:06 +02:00
Eugen Rochko 9bd23dc4e5
Track trending tags (#7638)
* Track trending tags

- Half-life of 1 day
- Historical usage in daily buckets (last 7 days stored)
- GET /api/v1/trends

Fix #271

* Add trends to web UI

* Don't render compose form on search route, adjust search results header

* Disqualify tag from trends if it's in disallowed hashtags setting

* Count distinct accounts using tag, ignore silenced accounts
2018-05-27 21:45:30 +02:00
Eugen Rochko 63c7b91572 Validate that e-mail resolves with MX and it's not blacklisted (#7631)
Original patch by @j-a4
2018-05-27 11:58:08 +09:00
Eugen Rochko 8378b72eba
Ensure push subscription is immediately removed when application is revoked (#7548)
* Ensure push subscription is immediately removed when application is revoked

* When token is revoked from app, unsubscribe too
2018-05-19 21:05:08 +02:00
Eugen Rochko 4b94e9c65e
Improve payload format of Web Push API now that it's open (#7521)
> Good lord what is happening in there

Previously the contents of the Web Push API payloads closely resembled the structure of JavaScript's [Notification](https://developer.mozilla.org/en-US/docs/Web/API/Notification). But now that the API is open to non-browser apps, and given that there is no required coupling between contents of the payload and a Notification object, here is how I changed the payload:

```json
{ 
  "access_token": "...",
  "preferred_locale": "en",
  "notification_id": "12345",
  "notification_type": "follow",
  "title": "So and so followed you",
  "body": "This is my bio",
  "icon": "https://example.com/avatar.png"
}
```

The title, body and icon attributes are included as a fallback so you can construct a minimal notification if you cannot perform a network request to the API to get more data.
2018-05-19 14:46:47 +02:00
Eugen Rochko 1e02dc8715
Add preference to hide following/followers lists (#7532)
* Add preference to hide following/followers lists

- Public pages
- ActivityPub collections (does not return pages but does give total)
- REST API (unless it's your own) (does not federate)

Fix #6901

* Add preference

* Add delegation

* Fix issue

* Fix issue
2018-05-18 02:26:51 +02:00
Eugen Rochko f31e58af9e
Fix nil error in StatusFilter (#7470)
Fix #7462
2018-05-13 20:26:30 +02:00
ThibG be7eeb7856 Catch Paperclip processing failures (fixes #6378) (#7439) 2018-05-11 13:14:50 +02:00
Eugen Rochko b4fb766b23
Add REST API for Web Push Notifications subscriptions (#7445)
- POST /api/v1/push/subscription
- PUT /api/v1/push/subscription
- DELETE /api/v1/push/subscription
- New OAuth scope: "push" (required for the above methods)
2018-05-11 11:49:12 +02:00
Eugen Rochko 42cd363542
Bot nameplates (#7391)
* Store actor type in database

* Add bot nameplate to web UI, add setting to preferences, API, AP
Fix #7365

* Fix code style issues
2018-05-07 09:31:07 +02:00
Eugen Rochko 61a9018607
Enable custom emojis in profiles (notes, field values, display names) (#7374)
Follow-up to #6124
2018-05-06 11:48:51 +02:00
Eugen Rochko 8da4bf0f90
4 profile fields max, store only 255 characters per name/value (#7348)
Fix #7303
2018-05-05 21:11:19 +02:00
Eugen Rochko c73ce7b695
Store home feeds for 7 days instead of 14 (#7354)
* Store home feeds for 7 days instead of 14

Reduces workload for status fan-out to active followers

* Fix test for user model
2018-05-05 00:54:24 +02:00
Eugen Rochko 6793bec4c6
Store URIs of follows, follow requests and blocks for ActivityPub (#7160)
Same URI passed between follow request and follow, since they are
the same thing in ActivityPub. Local URIs are generated during
creation using UUIDs and are passed to serializers.
2018-05-04 21:14:34 +02:00
ThibG a24605961a Fixes/do not override timestamps (#7336)
* Revert "Fixes/do not override timestamps (#7331)"

This reverts commit 581a5c9d29.

* Document Snowflake ID corner-case a bit more

Snowflake IDs are used for two purposes: making object identifiers harder to
guess and ensuring they are in chronological order. For this reason, they
are based on the `created_at` attribute of the object.

Unfortunately, inserting items with older snowflakes IDs will break the
assumption of consumers of the paging APIs that new items will always have
a greater identifier than the last seen one.

* Add `override_timestamps` virtual attribute to not correlate snowflake ID with created_at
2018-05-03 23:02:46 +02:00
Eugen Rochko a5293fdf61
Fix n+1 queries in StatusThreadingConcern (#7321) 2018-05-03 10:41:58 +02:00
Eugen Rochko a3d84e705a
Fix cache_associated no longer working (#7320) 2018-05-03 10:41:41 +02:00
abcang 71a7cea73f Keep notification when muting_notifications is true (#7311)
* Keep notification when muting_notifications is true

* Retrun mute object

* Fix test
2018-05-02 16:14:51 +02:00
Eugen Rochko cae933510c
Allow updating bio fields via PUT /api/v1/accounts/update_credentials (#7288)
Add raw bio fields to the source attribute on GET /api/v1/accounts/verify_credentials
2018-05-02 15:57:37 +02:00
Eugen Rochko c5dcd7d836
Speed up test suite by not generating RSA keys in test environment (#7296)
One RSA keypair for all fabricated test accounts is enough
2018-05-02 15:45:24 +02:00
Eugen Rochko a872392cd9
Add entity cache (#7271)
* Add entity cache

Use a caching layer for mentions and custom emojis that are
dynamically extracted from text.

Reduce duplicate text extractions

* Fix code style issue
2018-04-27 01:38:10 +02:00
Emelia Smith 60b871d56c Implement the ability for instances to define a list of disallowed hashtags (#7176)
The goal here isn't to prevent these hashtags from existing, but just to strongly curtail their usage; The hashtags may still exist in the database via federated status, or from being created prior to this feature.
2018-04-23 23:52:58 +02:00
Emelia Smith 495303d9b8 Prevent suspended accounts from appearing in AccountSearchService (#7246) 2018-04-23 21:27:18 +02:00
Akihiko Odaki 1258efa882 Paginate descendant statuses in public page (#7148) 2018-04-23 19:27:35 +02:00
Yamagishi Kazutoshi 9613a53cb3 Update dependencies for Ruby (2018-04-23) (#7237)
* Update annotate to version 2.7.3

* Update aws-sdk-s3 to version 1.9.2

* Update browser to version 2.5.3

* Update capistrano to version 3.10.2

* Update domain_name to version 0.5.20180417

* Update http to version 3.2.0

* Update lograge to version 0.10.0

* Update oj to version 3.5.1

* Update parallel_tests to version 2.21.3

* Update puma to version 3.11.4

* Update rubocop to version 0.55.0

* Update scss_lint to version 0.57.0

* Update simplecov to version 0.16.1

* Update tty-command to version 0.8.0

* Update tty-prompt to version 0.16.0

* Update pkg-config to version 1.3.0

* Update fog-local to version 0.5.0

* Update fog-openstack to version 0.1.25

* Update devise-two-factor to version 3.0.3

* bundle update
2018-04-23 11:29:17 +02:00
Eugen Rochko 7db7d68136
Detect and prevent image bombs, max. processable dimension 4096^2 (#7229) 2018-04-23 09:16:38 +02:00
Eugen Rochko 4ca2f73b12
Rescue Mastodon::LengthValidationError in Remoteable (#7228)
Fix #7198 by allowing records with optional attachments to save
2018-04-22 15:42:00 +02:00
Eugen Rochko bfe26ef67b
Force convert to JPG for preview card thumbnails to avoid animations (#7109)
* Force convert to JPG for preview card thumbnails to avoid animations

Fix #7093

* Conditionally convert to JPG only if original is GIF
Coalesce and strip on all formats to ensure no animated APNGs
2018-04-21 21:34:36 +02:00
Kaito Sinclaire 156b916caf Direct messages column (#4514)
* Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL

* Changes to match other timelines in 2.0
2018-04-18 13:09:06 +02:00
Eugen Rochko bb58fc003b
Fix warning about using SQL in order for Account#partitioned (#7159) 2018-04-17 13:50:48 +02:00
Eugen Rochko 78ed4ab75f
Add bio fields (#6645)
* Add bio fields

- Fix #3211
- Fix #232
- Fix #121

* Display bio fields in web UI

* Fix output of links and missing fields

* Federate bio fields over ActivityPub as PropertyValue

* Improve how the fields are stored, add to Edit profile form

* Add rel=me to links in fields

Fix #121
2018-04-14 12:41:08 +02:00
Yamagishi Kazutoshi 50529cbceb Upgrade Rails to version 5.2.0 (#5898) 2018-04-12 14:45:17 +02:00
Akihiko Odaki 519119f657 Paginate ancestor statuses in public page (#7102)
This also limits the statuses returned by API, but pagination is not
implemented in Web API yet. I still expect it brings user experience
better than making a user wait to fetch all ancestor statuses and flooding
the column with them.
2018-04-11 12:35:09 +02:00
Emelia Smith d9b62e34da Feature: Improve reports ui (#7032)
* Further improvements to Reports UI

- Clean up notes display
- Clean up add new note form
- Simplify controller
- Allow reopening a report with a note
- Show created at date for reports
- Fix report details table formatting

* Show history of report using Admin::ActionLog beneath the report

* Fix incorrect log message when reopening a report

* Implement fetching of all ActionLog items that could be related to the report

* Ensure adding a report_note updates the report's updated_at

* Limit Report History to actions that happened between the report being created and the report being resolved

* Fix linting issues

* Improve report history builder

Thanks @gargron for the improvements
2018-04-10 20:27:59 +02:00
Paul Woolcock 8f800ad691 Change custom emoji search to `ILIKE` instead of `=` (#7099) 2018-04-10 15:46:27 +02:00
Emelia Smith 219a4423d8 Feature: Allow staff to change user emails (#7074)
* Admin: Show unconfirmed email address on account page

* Admin: Allow staff to change user email addresses

* ActionLog: On change_email, log current email address and new unconfirmed email address
2018-04-10 09:16:06 +02:00
MIYAGI Hikaru e6e93ecd8a Fix GIFV encoding params (#7098)
- Explicitly specify video codec.
  When ffmpeg isn't compiled with libx264 but openh264, mpeg4 is selected as video codec.
- Swap avarage bitrate and max bitrate.
2018-04-10 09:11:55 +02:00
Akihiko Odaki 498327b2e3 Exclude status itself from context query (#7083)
ancestor_statuses and descendant_statuses used to include the root status
itself, but the behavior is confusing because the root status is not
an ancestor nor descendant.
2018-04-09 09:58:53 +02:00
Emelia Smith e85cffb236 Feature: Report improvements (#6967) (#7000)
* Implement Assignment of Reports (#6967)

* Change translation of admin.report.comment.label to "Report Comment" for clarity

As we'll soon add the ability for reports to have comments on them, this clarification makes sense.

* Implement notes for Reports

This enables moderators to leave comments about a report whilst they work on it

* Fix display of report moderation notes

* Allow reports to be reopened / marked as unresolved

* Redirect to reports listing upon resolution of report

* Implement "resolve with note" functionality

* Add inverse relationship for report notes

* Remove additional database querying when loading report notes

* Fix tests for reports

* Fix localisations for report notes / reports
2018-04-02 22:04:14 +02:00
David Underwood 123a343d11 [WIP] Enable custom emoji on account pages and in the sidebar (#6124)
Federate custom emojis with accounts
2018-04-01 23:55:42 +02:00
Eugen Rochko 2a90da1837
Fix UniqueUsernameValidator comparison (#6926)
Comparison was downcasing only one side, therefore if previously
existing account had a non-lowercase spelling, it would be ignored
when checking for duplicates.

New rake task `mastodon:maintenance:find_duplicate_usernames` will
help find constraint violations that might have occured from the
presence of this bug.

Bump version to 2.3.3
2018-03-27 04:33:57 +02:00
Akihiko Odaki 40e5d2303b Validate HTTP response length while receiving (#6891)
to_s method of HTTP::Response keeps blocking while it receives the whole
content, no matter how it is big. This means it may waste time to receive
unacceptably large files. It may also consume memory and disk in the
process. This solves the inefficency by checking response length while
receiving.
2018-03-26 14:02:10 +02:00
Akihiko Odaki b2a4ffd3a9 Change columns in notifications nonnullable (#6764) 2018-03-24 12:51:28 +01:00
Akihiko Odaki 54b273bf99 Close http connection in perform method of Request class (#6889)
HTTP connections must be explicitly closed in many cases, and letting
perform method close connections makes its callers less redundant and
prevent them from forgetting to close connections.
2018-03-24 12:49:54 +01:00
Eugen Rochko da70aca28e
Restore username validation to disallow dots, for now (#6863)
Usernames with dots in them do not work with routes, because the dot usually separates the desired page format (e.g. json). I don't want to mess with changing route constraints for this patch release.
2018-03-22 11:30:22 +01:00
ThibG 6f531d140b Fix MENTION_RE to not match nil usernames (#6862) 2018-03-22 10:45:48 +01:00
Eugen Rochko 93897134ca
Permit dots in usernames with conditions (#6844)
* Permit dots in usernames with conditions

- Dot cannot be the start or end of username
- a.lice and al.ice are considered the same during sign-up

* Fix regex mixin flags
2018-03-21 10:26:53 +01:00
Alexander 0306e3e9be bugfixes and gem update (#6831)
* update to new version of devise_pam_authenticatable2

* fix behaviour if suffix is nil, fix environment loading, fix user email creation

* code cleanup/fix linter warning
2018-03-19 20:08:56 +01:00
Daniel Hunsaker 4a0a19fe54 Handle Mastodon::HostValidationError when pulling remoteable assets (#6782)
This will prevent, for example, `rake mastodon:redownload_avatars` from crashing when an instance is no longer responding to connection attempts, instead silently continuing as expected.
2018-03-17 13:27:50 +01:00
Eugen Rochko 1aaec701bb
Fix #6715: Make catalan words with the L geminate letter work in hashtags (#6741) 2018-03-11 14:55:49 +01:00
Renato "Lond" Cerqueira 9aba44ea79 Rescue when there's no extension in the remotable (#6358)
* Rescue when there's no extension in the remotable
Sometimes the remotable is pointing to a directory with no file
extension. Maybe it should not be expecting to identify based on
extensions to begin with, but since it's the case, it should be ready
for it.

* Fix codeclimate issue

* Check if filename is nil instead of rescueing exception
Suggestion made in the PR

* Avoid concatenation issue if filename is nil
If filename is nil, extname was undefined

* Invert condition
Address PR comments
2018-03-08 02:25:10 +01:00
Eugen Rochko cfa9b6e13a
Remove text requirement when media attached from statuses (#6672) 2018-03-07 08:28:52 +01:00
Effy Elden dd9d00d293 Add additional first_name and last_name SAML attribute statement options, and modify Omniauthable concern to use full_name or first_name + last_name if not available (#6669) 2018-03-07 06:19:10 +01:00
Eugen Rochko 78d772af86
Fix #3807: Increase avatars to 400x400 max (#6651)
But do not upscale when they are smaller
2018-03-06 06:29:01 +01:00
Akihiko Odaki c110fa62ac Provide default OTP_SECRET value for development environment (#6617) 2018-03-04 20:28:24 +01:00
Eugen Rochko 9110db41c5
Federate pinned statuses over ActivityPub (#6610)
* Federate pinned statuses over ActivityPub

* Display pinned toots in web UI

Fix #6117

* Fix migration

* Fix tests

* Update outbox_serializer.rb

* Update remove_serializer.rb

* Update add_serializer.rb

* Update fetch_featured_collection_service.rb
2018-03-04 09:19:11 +01:00
Eugen Rochko 47bdb9b33b
Fix #942: Seamless LDAP login (#6556) 2018-02-28 19:04:53 +01:00
Eugen Rochko 41a01bec23
Federated reports (#6570)
* Fix #2176: Federated reports

* UI for federated reports

* Add spec for ActivityPub Flag handler

* Add spec for ReportService
2018-02-28 06:54:55 +01:00
Ian McCowan c33931b613 Fix prev/next links on public profile page (#6497)
* Fix prev/next links on public profile page

* Don't make pagination urls if no available statuses

* Fix empty check method

* Put left chevron before prev page link

* Add scope for pagination "starting at" a given id

* Status pagination try 2:

s/prev/older and s/next/newer
"older" on left, "newer" on right
Use new scope for "newer" link
Extract magic 20 page size to constant
Remove max_id from feed pagination as it's not respected

* Reinstate max_id for accounts atom stream

* normalize
2018-02-26 03:31:28 +01:00
Eugen Rochko 5cc716688a
Ensure the app does not even start if OTP_SECRET is not set (#6557)
* Ensure the app does not even start if OTP_SECRET is not set

* Remove PAPERCLIP_SECRET (it's not used by anything, actually)

Imports are for internal consumption and the url option isn't even
used correctly, so we can remove the hash stuff from them
2018-02-26 01:31:44 +01:00
Ghislain Loaec e668180044 New variable OAUTH_REDIRECT_AT_SIGN_IN + Ref #6538 (not only SAML strategies) (#6540) 2018-02-23 01:16:17 +01:00
Ghislain Loaec 3084fe4959 New env variable: SAML_SECURITY_ASSUME_EMAIL_IS_VERIFIED + fixes #6533 (#6538) 2018-02-22 23:31:25 +01:00
Eugen Rochko 90f12f2e5a
Focal points (#6520)
* Add focus param to media API, center thumbnails on focus point

* Add UI for setting a focal point

* Improve focal point icon on upload item

* Use focal point in upload preview

* Add focalPoint property to ActivityPub

* Don't show focal point button for non-image attachments
2018-02-22 00:35:46 +01:00
Eugen Rochko d3a62d2637
Fix #6525: Make sure file is opened in LazyThumbnail processor (#6529) 2018-02-22 00:28:19 +01:00
Eugen Rochko 61ed133fea
Account archive download (#6460)
* Fix #201: Account archive download

* Export actor and private key in the archive

* Optimize BackupService

- Add conversation to cached associations of status, because
  somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
  (Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
  (Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
  (Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
  GIF-to-MP4 media are all named "media.mp4" for some reason

* Keep uniquely generated filename in Paperclip::GifTranscoder

* Ensure dumped files do not overwrite each other by maintaing directory partitions

* Give tar archives a good name

* Add scheduler to remove week-old backups

* Fix code style issue
2018-02-21 23:21:32 +01:00
Eugen Rochko a7171af0a3
Fix avatar and header issues by using custom geometry detector (#6515)
* Fix avatar and header issues by using custom geometry detector

Revert a part of #6508. The file passed to dynamic styles method
was not actually a file, but an instance of Paperclip::Attachment,
which broke all styles by always returning {} from the method.

One problem with GIF avatars was that Paperclip::GeometryDetector
reported wrong dimensions for them, e.g. 120x120 GIF avatar would
for some reason be detected as 120x53. By writing our own geometry
parser, we can use FastImage, which also happens to be faster than
ImageMagick, to detect image dimensions, which are also correct.

Unfortunately, this PR does not implement skipping a `convert`
entirely if the dimensions are already correct, as I found no easy
way to write that behaviour into Paperclip without rewriting the
Paperclip::Thumbnail class.

* Only invoke convert if dimension or format needs to be changed
2018-02-21 03:40:12 +01:00
Eugen Rochko 66105929e0
Don't resize avatars/headers if their dimensions are already right (#6508)
Also don't apply "-quality 80" option which is probably the reason
for slight color differences between original and remote image
(because it would apply it twice, once on original instance, and
again on the receiving instance)
2018-02-19 16:06:12 +01:00
Eugen Rochko cba2897108
Cache relationships in API (#6482)
* Cache relationships in API

* Fetch relationships for search results in UI

* Only save one account's maps in each cache item
2018-02-18 03:14:46 +01:00
Eugen Rochko a71af98401
Push discovered status through streaming API within a time window (#6484)
Time window of 6 hours
2018-02-17 14:28:48 +01:00
Eugen Rochko 9dbae6e8a1
Save video metadata and improve video OpenGraph tags (#6481)
* Save metadata from video attachments, put correct dimensions into OG tags

* Add twitter:player for videos

* Fix code style and test
2018-02-16 07:22:20 +01:00
Eugen Rochko 3ebc0ad4d3
Full-text search for authorized statuses (#6423)
* Add full-text search for authorized statuses

- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index

Fix #5880
Fix #4293
Fix #1152

* Add commented out docker-compose configuration for ES container

* Optimize index import, filter search results

* Add basic normalization to the index

* Add better stemming and normalization to the index

* Skip webfinger request if search query includes both @ and a space

* Fix code style

* Visually separate search result sections

* Fix code style issues
2018-02-09 23:04:47 +01:00
Eugen Rochko 76f3d5d16b
Add preference to always display sensitive media (#6448) 2018-02-09 00:26:57 +01:00
abcang 298c81c00f Clear account cache of notification target_status (#6442) 2018-02-08 15:33:23 +01:00
Akihiko Odaki 67f7ffa792 Change user_id column non-nullable (#6435) 2018-02-07 16:35:44 +01:00
Eugen Rochko 38e0133e1b
Make PAM gem optional, allow configuration over environment (#6415) 2018-02-04 15:05:53 +01:00
Renato "Lond" Cerqueira 4e4f1b0dcb Add option to show only local toots in timeline preview (#6292)
* Add option to show only local toots in timeline preview
Right know, toots from all the known fediverse are shown in the main
page of an instance. That however doesn't reflect the instance itself.
With this option the admin may choose to display only local toots so
that users checking the instance get a better idea of internal
conversations.

* Fix issues pointed by codeclimate and eslint

* Add default message for community timeline

* Update pl.yml
2018-02-04 06:00:10 +01:00