Commit Graph

274 Commits

Author SHA1 Message Date
Eugen Rochko 2dd30804b6
Change how unconfirmed accounts are displayed in admin UI (#17874)
Fix #17815
2022-03-26 02:53:13 +01:00
Eugen Rochko edf09ec747
Add `/api/v1/accounts/familiar_followers` to REST API (#17700)
* Add `/api/v1/accounts/familiar_followers` to REST API

* Change hide network preference to be stored consistently for local and remote accounts

* Add dummy classes to migration

* Apply suggestions from code review

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-07 09:36:47 +01:00
Eugen Rochko 27965ce5ed
Add trending statuses (#17431)
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
Eugen Rochko 564efd0651
Add appeals (#17364)
* Add appeals

* Add ability to reject appeals and ability to browse pending appeals in admin UI

* Add strikes to account page in settings

* Various fixes and improvements

- Add separate notification setting for appeals, separate from reports
- Fix style of links in report/strike header
- Change approving an appeal to not restore statuses (due to federation complexities)
- Change style of successfully appealed strikes on account settings page
- Change account settings page to only show unappealed or recently appealed strikes

* Change appealed_at to overruled_at

* Fix missing method error
2022-02-14 21:27:53 +01:00
Claire 0a120d86d2
Fix error-prone SQL queries (#15828)
* Fix error-prone SQL queries in Account search

While this code seems to not present an actual vulnerability, one could
easily be introduced by mistake due to how the query is built.

This PR parameterises the `to_tsquery` input to make the query more robust.

* Harden code for Status#tagged_with_all and Status#tagged_with_none

Those two scopes aren't used in a way that could be vulnerable to an SQL
injection, but keeping them unchanged might be a hazard.

* Remove unneeded spaces surrounding tsquery term

* Please CodeClimate

* Move advanced_search_for SQL template to its own function

This avoids one level of indentation while making clearer that the SQL template
isn't build from all the dynamic parameters of advanced_search_for.

* Add tests covering tagged_with, tagged_with_all and tagged_with_none

* Rewrite tagged_with_none to avoid multiple joins and make it more robust

* Remove obsolete brakeman warnings

* Revert "Remove unneeded spaces surrounding tsquery term"

The two queries are not strictly equivalent.

This reverts commit 86f16c537e06c6ba4a8b250f25dcce9f049023ff.
2022-01-23 18:10:10 +01:00
Eugen Rochko 8e84ebf0cb
Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
Eugen Rochko 0fb9536d38
Add batch suspend for accounts in admin UI (#17009) 2021-12-05 21:48:39 +01:00
Takeshi Umeda 3419d3ec84
Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915)
* Bump chewy from 5.2.0 to 7.2.2

* fix style (codeclimate)

* fix style

* fix style

* Bump chewy from 7.2.2 to 7.2.3
2021-11-18 22:02:08 +01:00
Claire fc3ae1343d
Switch from unmaintained paperclip to kt-paperclip (#16724)
* Switch from unmaintained paperclip to kt-paperclip

* Drop some compatibility monkey-patches not required by kt-paperclip

* Drop media spoof check monkey-patching

It's broken with kt-paperclip and hopefully it won't be needed anymore

* Fix regression introduced by paperclip 6.1.0

* Do not rely on pathname to call FastImage

* Add test for ogg vorbis file with cover art

* Add audio/vorbis to the accepted content-types

This seems erroneous as this would be the content-type for a vorbis stream
without an ogg container, but that's what the `marcel` gem outputs, so…

* Restore missing for_as_default method

* Refactor Attachmentable concern and delay Paperclip's content-type spoof check

Check for content-type spoofing *after* setting the extension ourselves, this
fixes a regression with kt-paperclip's validations being more strict than
paperclip 6.0.0 and rejecting some Pleroma uploads because of unknown
extensions.

* Please CodeClimate

* Add audio/vorbis to the unreliable set

It doesn't correspond to a file format and thus has no extension associated.
2021-09-29 23:52:36 +02:00
Claire 2b18f7a943
Fix processing mentions to domains with non-ascii TLDs (#16689)
Fixes #16602
2021-09-01 22:06:40 +02:00
Claire 5efb1ff337
Fix followers synchronization mechanism not working when URI has empty path (#16510)
* Fix followers synchronization mechanism not working when URI has empty path

To my knowledge, there is no current implementation on the fediverse
that can use bare domains (e.g., actor is at https://example.org instead of
something like https://example.org/actor) that also plans to support the
followers synchronization mechanism. However, Mastodon's current implementation
would exclude such accounts from followers list.

Also adds tests and rename them to reflect the proper method names.

* Move url prefix regexp to its own constant
2021-08-11 17:48:42 +02:00
Claire 13b08610a0
Fix crash when encountering invalid account fields (#16598)
* Add test

* Fix crash when encountering invalid account fields
2021-08-11 16:40:55 +02:00
Claire 30ce6e395c
Fix user email address being banned on self-deletion (#16503)
* Add tests

* Fix user email address being banned on self-deletion

Fixes #16498
2021-07-14 05:35:49 +02:00
Eugen Rochko 38b3419950
Fix not being able to suspend accounts that already have a canonical e-mail block (#16455) 2021-07-02 12:03:15 +02:00
Eugen Rochko 1294f9ee4f
Remove PubSubHubbub-related columns from accounts table (#16170) 2021-05-07 19:32:58 +02:00
Eugen Rochko 74081433d0
Change trending hashtags to be affected be reblogs (#16164)
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending

Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots

Remove references to hashtags in profile directory from the code
and the admin UI
2021-05-07 14:33:43 +02:00
Eugen Rochko b3ceb3dcc4
Add canonical e-mail blocks for suspended accounts (#16049)
Prevent new accounts from being created using the same underlying
e-mail as a suspended account using extensions and period
permutations. Stores e-mails as a SHA256 hash
2021-04-17 03:14:25 +02:00
Eugen Rochko f7117646af
Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Claire 5cc45d22d3
Remove subscription_expires_at leftover from OStatus (#15857) 2021-03-12 05:25:24 +01:00
ThibG 54d4e5252b
Use Rails' index_by where it makes sense (#15542)
* Use Rails' index_by where it makes sense

* Fix tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-12 09:27:38 +01:00
luigi 087ed84367
Optimize map { ... }.compact calls (#15513)
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
2021-01-10 00:32:01 +01:00
abcang efffdd3778
Fix rubocop config and warnings (#15503)
* disable NewCops

* update TargetRubyVersion

* Fix Lint/MissingSuper for ActiveModelSerializers::Model

* Fix Lint/MissingSuper for feed

* Fix Lint/FloatComparison

* Do not use instance variables
2021-01-07 09:40:55 +01:00
ThibG 3249d35bdc
Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
ThibG 8357969559
Fix admins being able to suspend their instance actor (#14567)
* Fix admin being able to suspend their own instance account

* Add text about the instance's own actor in admin view

* Change instance actor notice from flash message to template

* Do not list local instance actor in account moderation list
2020-12-15 17:23:58 +01:00
Eugen Rochko 216b85b053
Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
Eugen Rochko 48bef17cc9
Fix slow distinct queries where grouped queries are faster (#15287)
About 2x speed-up on inboxes query
2020-12-07 12:08:30 +01:00
Eugen Rochko f844386809
Add `tootctl accounts merge` (#15201)
* Add `tootctl accounts merge`

* Update lib/mastodon/accounts_cli.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-11-23 17:50:16 +01:00
Eugen Rochko 3134691948
Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
Takeshi Umeda d6fe0c94ca
Add account sensitized (#14361)
* Add account sensitized

* Fix i18n normalize

* Fix description and spec

* Fix spec

* Fix wording
2020-11-04 20:45:01 +01:00
ThibG ca56527140
Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
Eugen Rochko ed099d8bdc
Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
Eugen Rochko 5d8398c8b8
Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
Eugen Rochko c3ca3801f2
Add separate cache directory for non-local uploads (#12821) 2020-04-26 23:29:08 +02:00
Eugen Rochko 5edff32733
Change delivery failure tracking to work with hostnames instead of URLs (#13437) 2020-04-15 20:33:24 +02:00
ThibG b154428e14
Add federation support for the "hide network" preference (#11673)
* Change ActivityPub follower/following collections to not link first page

* Add support for hiding followers and following of remote users

* Switch to using a single `hide_collections` column

* Address code style remarks
2020-03-09 00:10:29 +01:00
ThibG 4a4cd686c1
Add sorting by username, creation and last activity in moderation view (#13076)
* Add ability to order accounts in moderation view

* Display last status date in “Most recent activity” for remote users
2020-03-08 15:39:13 +01:00
ThibG 7b63c5469e
Fix announcements with fully-qualified mention to local user crashing WebUI (#13164) 2020-02-28 13:49:45 +01:00
abcang 61a7390b66
Search account domain in lowercase (#13016)
* Search account domain in lowercase

* fix rubocop error

* fix spec/models/account_spec.rb
2020-02-01 15:42:24 +01:00
Eugen Rochko f52c988e12
Add announcements (#12662)
* Add announcements

Fix #11006

* Add reactions to announcements

* Add admin UI for announcements

* Add unit tests

* Fix issues

- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"

* Fix scheduler unpublishing announcements before they are due

* Fix filter params not being passed to announcements filter
2020-01-23 22:00:13 +01:00
ThibG 57e2833f6a Remove dependency on OStatus2 gem (#12822) 2020-01-11 21:36:53 +01:00
Eugen Rochko f86ee4b59f
Fix IDN mentions not being processed, IDN domains not being rendered (#12715)
This changes the REST API to return unicode domains in the `acct`
attribute instead of punycode, and to render unicode instead of
punycode on public HTML pages as well.

Fix #7812, fix #12246
2019-12-30 19:20:43 +01:00
Takeshi Umeda f43f1e0184 Add basic support for group actors (#12071)
* Show badge on group actor in WebUI

* Do not notify in case of  by following group actor

* If you mention group actor, also mention group actor followers

* Relax characters that can be used in username (same as Application)

* Revert "Relax characters that can be used in username (same as Application)"

This reverts commit 7e10a137b878d0db1b5252c52106faef5e09ca4b.

* Delete display_name method
2019-12-04 20:36:33 +01:00
ThibG 2edf6d81cb Simplify SQL query when performing account search amongst followings (#12302) 2019-11-21 17:35:04 +01:00
ThibG 65e13cfacf Add abilityto add oneself to lists (#12271)
* Add ability to add oneself to lists

* Change search results to include oneself when searching through followers

* Mark follow relation as optional in ListAccount
2019-11-04 13:02:01 +01:00
Eugen Rochko b5f7e12817
Remove auto-silence behaviour from spam check (#12117)
Fix #12113
2019-10-09 07:11:23 +02:00
Eugen Rochko 354fdd317e
Fix attachment not being re-downloaded even if file is not stored (#12125)
Change the behaviour of remotable concern. Previously, it would skip
downloading an attachment if the stored remote URL is identical to
the new one. Now it would not be skipped if the attachment is not
actually currently stored by Paperclip.
2019-10-09 07:10:46 +02:00
Eugen Rochko 5c42f47617
Fix records not being indexed sometimes (#12024)
It's possible that after commit callbacks were not firing when
exceptions occurred in the process. Also, the default Sidekiq
strategy does not push indexing jobs immediately, which is not
necessary and could be part of the issue too.
2019-10-01 01:19:11 +02:00
Eugen Rochko c5d37f18cb
Change deletes to preserve soft-deleted statuses in unresolved reports (#11805)
Change all account actions except "none" to resolve all unresolved reports

Refactor `SuspendAccountService` to be more readable
2019-09-11 16:32:44 +02:00
Eugen Rochko cb447b28c4
Add profile directory to web UI (#11688)
* Add profile directory to web UI

* Add a line of bio to the directory
2019-08-30 00:14:36 +02:00
Eugen Rochko 8fdff2748f
Add more accurate account search (#11537)
* Add more accurate account search

When ElasticSearch is available, a more accurate search is implemented:

- Using edge n-gram index for acct and display name
- Using asciifolding and cjk width normalization on display names
- Using Gaussian decay on account activity for additional scoring (recency)
- Using followers/friends ratio for additional scoring (spamminess)
- Using followers number for additional scoring (size)

The exact match precedence only takes effect when the input conforms
to the username format and the username part of it is complete, i.e.
when the user started typing the domain part.

* Support single-letter usernames

* Fix tests

* Fix not picking up account updates

* Add weights and normalization for scores, skip zero terms queries

* Use local counts for accounts index, adjust search parameters

* Fix mistakes

* Using updated_at of accounts is inadequate for remote accounts
2019-08-16 01:24:03 +02:00