morbane | AO3 collections moderation queue "bug"

This is a conversation I'm having with AO3's AD&T, through Support; I'm posting it here because the outcome may be of interest to other AO3 gift exchange moderators. tl;dr: there is a thing, relatively minor, that I don't want AO3 to do, and I am attempting to further argue the case.

Obviously, this post has an agenda. I am hoping other moderators will also say, to AO3, please don't do the thing. There is a difference between "Morbane gets a bugbear; these are the wishes of one user" and "Multiple users have demonstrated the utility of a function in several different use cases" - not that I expect AO3 to ignore the wishes of one user, but I expect them to prioritise the latter more. But, if other moderators mind less, that is also useful to know.

The global settings and moderation queue of an AO3 collection: how they work
An AO3 collection can be set to "revealed" or "unrevealed" (works are available for reading, or not), and, independently, "not anonymous" and "anonymous" (works do, or don't, display their creator's names). A collection can also be set to moderated or unmoderated.

When a collection is unmoderated, any work submitted to that collection appears in a list of Approved Items. Any change made to the settings of the collection affects the Approved Items. If a work is submitted when the collection is unrevealed, and the collection is then set to revealed, the work will become available to read.

When a collection is moderated, any* work submitted to that collection appears in a moderation queue of items "Awaiting Approval", which the moderator can review before adding works to the Approved list as they choose. (There's also a Rejected list.)

Whether a collection is moderated or unmoderated, a moderator can select any Approved or Rejected item and give them an unapproved setting so that they go to the Awaiting Approval list.

Specifically...
For several years - I forget when I first used this function - the Awaiting Approval moderation queue has seemed particularly useful, because an item left in this queue does not reveal or have its creator name appear when those settings are changed in the rest of the collection.

However, when we asked AD&T AO3 staff to support the Yuletide mods and watch the collection back-end at the moment of revealing Yuletide stories - which they did, for which we thank them - we discovered that the moderation queue isn't supposed to work this way. Any item submitted (or, presumably, invited) to a collection, whether awaiting approval, rejected, or approved, should apparently obey every setting change made to the collection. When the works are revealed, that should apparently include the rejected and un-approved works. When the creator's names are revealed, that should again apparently include the rejected and un-approved works. The only difference is that the rejected and un-approved works will not appear in the collection itself or display the collection link anywhere.

I am, however, frustrated that AO3 wants to fix this behaviour - and indicated that this was a matter of some urgency to them - because the ability to corral off certain selected works and keep them from going 'live' is a really useful tool for quarantining spitefic, picklefic, unfinished works, duplicated works, and in some past cases, especially egregious Do-Not-Want fills. There are works that moderators do not want to reveal.

*except for works submitted by members of collections, which sail right past the moderation queue

Correspondence with Support to date
I wrote in a few days after Yuletide reveals when this mismatch of expectations came to light.

December 28, 2018

Dear Support and AD&T,

Yesterday the Yuletide moderating team brought to your attention a way in which the collections moderation queue works. AO3 staff apparently categorize this as a bug; I categorize it as a desired behaviour that is both reliable and relied on. I am writing to ask you to please not change this behaviour.

Heretofore, when:
-a collection is set to unrevealed and anonymous,
-and some works are approved and some are in the moderation queue,
-and the moderator then changes the collection settings to revealed or revealed and non-anonymous,
-the result has been that the works in the moderation queue do not pick up the new settings, but stay unrevealed.

I believe that this behaviour has been around long before the "reveals bug" in which works may fail to obey, or pick up, a collection's settings for other reasons.

It is a useful tool for moderators and one I have relied on in the past. Removing this functionality would cause me, and I believe other users, a range of negative outcomes including frustration and embarrassment.

For example: this year, a user submitted two versions of the same work, near-identical, to the collection. This was clearly a mistake, but they did not respond to emails suggesting they delete one of the works. Putting one of the two works in the moderation queue before reveals, in order to keep it from revealing, seemed like the most appropriate way to handle this. Our other options were to reveal it with the rest (which the coders supervising the Yuletide reveals inadvertently did), meaning kudos & comments split between the two works, and a disappointed recipient), or to reject** it, which would have given away the authorship of the work still in the collection, and kudos & comments split again. In any case, the recipient would have been confused, but if the work had stayed unrevealed, they would not have got doubled-up notifications and could have easily clarified with us why they had a Mystery Gift still.

In other cases, we have aimed to keep hidden
-works that are not real works, for example find-and-replace fics or plagiarized text from the internet, where we have told the authors that we will happily reveal the works if they update them with fic
-fic that contained heavy use of DNWs stated by the recip to be triggering (a fic featuring drug overdose and suicide where the recipient had clearly asked for NOT THAT).
-theoretically, spitefic***.

When we discussed this with you yesterday, you offered a workaround for our collection this year of setting it back to unrevealed and moderated, and putting the sensitive works back in the moderation queue where they would pick up the collection's new settings.

If this is going to be your suggestion going forward, then, please don't fix the "bug" - it's hard to see the point of allowing moderators the same effect but only if they achieve it in a more roundabout way. In order to get the effect we want to when revealing Yuletide next year, I imagine we will leave the collection on unrevealed and anonymous, and update the settings on every approved work, one by one, through over 100 pages of 20 works each. That seems punitive.

If you aim by fixing the bug to remove any ability by moderators to change a work's settings back to unrevealed or anonymous in any situation, once it has already revealed, please reconsider and seek further feedback. I am aware that there is some tension between the control an author should have over their work and the control a collection moderator should have in this situation, but an author can remove a work from a collection at any time.

Please clarify what functionality we are supposed to have. If I've been abusing my power I'd certainly like to know!

Please don't fix this bug.

Kind regards, Morbane

**I should have said 'remove' rather than 'reject' here.
***As it so happens we have not used this feature to keep any spitefic hidden - spitefic is generally discovered after a collection opens. If we happened to discover it early, we'd definitely want to hold it back, though.

December 30, 2018

Hi, Morbane:

Thanks for asking about this bug. I checked with the Coders on this. The bug is unintentional behavior by the Archive code that can trap works in an unrevealed state, leaving them unfindable, and thus needs to be fixed. While you're not doing so, this can be used in an abusive manner, and thus needs to be fixed. They intend to fix this bug by making it so that unapproved and rejected works become revealed when the collection becomes revealed. They also noted that they are testing a fix for works getting stuck in reveal, and plan to deploy it as soon as possible.

There is no plan to remove the ability for a collection moderator to put a collection or work back in an unrevealed status. We appreciate your feedback. If you have other thoughts, let us know.

Best,
CJ
AO3 Support co-chair

January 3, 2019

Dear CJ and team,

I appreciate the speed of this response and that you're very clear about coders' position here. Thank you.

It's also very exciting to have a formal confirmation that a fix for works being stuck at various settings is in the works. Thanks for telling me. I hope testing goes smoothly!

I'm still a little puzzled by this response - especially the position that the moderation queue behaviour before fixing is open to abuse, and yet, it is OK for the same effect to be achieved with more effort. That seems just as open to abuse as doing it simply. I am travelling at the moment and will comment further in a few days. Thank you for considering my feedback.

In the meantime, I have another question about the desired behaviour of the moderation functions for challenges, please. I'm wondering about the rejected from collection list. If a moderator has chosen to reject a work before changing a collection setting, is it your intention that the work picks up the new collection setting or not?

As far as I can tell, an author also gets no notification when a work has been rejected. The banner displayed on the work on the user's side only indicates that it is part of a collection and will be revealed soon. Is this intentional?

Thank you, Morbane

As I did not indicate any urgency to those follow-up questions, I have not yet heard back.

My chief issue is: this is a useful, reliable thing that works to help moderators do a thing they want to do. I'm sure it's possible to abuse it, but given it's worked this way for years and coders did not know that, I feel that reports of its abuse must have been rare to nonexistent.

And their fix does not make it impossible for me, or another moderator, to abuse it - it merely makes it hard, at the same time as making other things hard. If it's a power that can be abused and they want to do something about it, why leave me with that power? After their fix, I could still go mad with power and go back and hide a single work in each of Yuletide 2010, 2011, 2012, etc, etc. The poor author and recipient might not notice for months or years, and then be very dismayed when they did. If that's the kind of abuse they mean (and abuse it would surely be!), then the fix would in no way remove my ability to do it.

Thoughts and suggestions very welcome - including if you disagree with me! And if you're a moderator who also would like AO3 not to "fix" this "bug", please tell them so.

I received a further reply:
January 13, 2019

Hi,

Thank you for these concerns regarding this bug in the Collection feature. Because a number of users have submitted similar tickets, we are collating the information and sending the same response to all users.

While we are open to suggestions for better exchange moderation tools, we view the ability to leave unrevealed works in an otherwise revealed collection as a bug that has to be fixed, not a feature, because it can trap works in an unrevealed state when the work is unapproved. We note that this can be done accidentally, if the work's creator does not regularly check their works. We also note that this can be done maliciously, in the case of a moderator judging a participant's work as "not good enough". We have known about the bug for years, but they weren't aware it was being intentionally exploited. We're somewhat relieved to hear it's being used with good intentions but we can't count on it remaining that way as knowledge of the bug is spread. As well, tags used on these "hidden" works populate the wrangling queue, impeding Tag Wranglers' workflow. We want to note that, in the case of keeping the work in the moderation queue to prevent the recipient from getting a mismatched work, a malicious creator can still remove the work from the collection and directly gift it.

We suggest that if moderators retain the need to keep some works unrevealed, they should leave the Collection itself unrevealed and reveal individual works instead.

We want to note that the fix to this bug will likely be deployed in the vicinity of a number of other Collection and Challenge bug fixes, including fixing the issue where some works stay unrevealed when a Collection is revealed, and fixing an issue where the email notification for a rejected (and revealed) work would have the name of the collection it was rejected from. Most of these fixes have been in process for a while, but the Coders have had a backlog of more critical projects.

For your new question, once a work is rejected, it should be completely disconnected from the collection, and thus not pick up status changes. Do you have a link to a work that's misfiring?

Best,
CJ
AO3 Support co-chair

I'm pondering a reply; they do seem pretty set on this choice. I respectfully disagree that this is something unknown that will be more open to abuse as knowledge "spreads". I feel like a lot of moderators have known about it for a while and because we thought it was designed behaviour, we treated it responsibly. Of course, I don't know all cases. I do know of one exchange that stayed unrevealed for years because the mod essentially flaked. That seems unfortunate - but not a disaster - to me.

NOTE: In replies others have received, CJ suggests grouping multiple comments to this issue in single documents/emails to aid Support in processing comments on this issue. The original support ticket I was assigned is ## 206703 ##. If you send a new comment to AO3 about this, I suggest quoting that ticket in your subject line.

Reply, post-pondering
January 14, 2019

Dear CJ and team,

Thank you for your reply.

It's reassuring and helpful to hear that if/when this bug is fixed, it will be fixed as part of a suite of fixes including the reveals bug that struck at random to prevent works picking up intended collection settings. Since that's been a high-profile bug, this alleviates a concern I had that moderators who use the moderation queue "bug" would not know that it had been fixed and would be further inconvenienced by its fixing. Excellent. I wish coders and testers all luck with the reveals bug fix, and again, I'm grateful to receive an update on it.

It's also disquieting to hear that this will be part of a suite of unspecified fixes, because I already know that this suite includes a fix that, for me, is an act of breaking. I wonder what impact other fixes will have on me. I look forward to more information in due time.

To return to your last question,

For your new question, once a work is rejected, it should be completely disconnected from the collection, and thus not pick up status changes. Do you have a link to a work that's misfiring?

I think you mean removed, not rejected? I may have muddled this issue by using the wrong term at a point in my original submission - "to reject it, which would have given away the authorship of the work still in the collection". I meant removed there. Sorry. So that we're 100% on the same page:

-To remove a work from the collection is to immediately eject it; it will act as though it was not submitted to the collection but instead published outside it. I haven't tested this but I believe this would send a gift notification email if the collection were originally unrevealed, and the work were a gift. Happily, I do not know of any work that has been removed from a collection and is misfiring by still showing that collection's status changes.

-To reject a work from the collection is to add it to a list like Approved. This list is called Rejected Items. I am not sure what the function of this list is, and that's what I was asking about. Because of other things you have said, I'm wondering if this is meant to be a sort of pre-Remove list, where moderators put things they want to remove, but not yet? For example, maybe someone submits an unfinished work; according to coders, the moderator might then put it in the Reject list, and when revealing the collection, the moderator might go to the Rejected page and Remove all the rejected works, per page, at the same time. I am guessing, here. Is this how it's meant to work? Right now, a work in the Rejected list does not pick up a change made to global collection settings from Unrevealed to Revealed, or Anonymous to Non-Anonymous. I assume that this is another manifestation of the behaviour you consider a bug. Hence my questions:

If a moderator has chosen to reject a work before changing a collection setting, is it your intention that the work picks up the new collection setting or not?

As far as I can tell, an author also gets no notification when a work has been rejected. The banner displayed on the work on the user's side only indicates that it is part of a collection and will be revealed soon. Is this intentional?

You make the point that if this bug is not fixed, there is the potential for works to be trapped. I'd like to go into that a bit further.

Each year since exchange functionality was set up, thousands of works - maybe now tens of thousands - are trapped in an unrevealed state. This is not a flaw: this is on purpose. This is what happens when a user voluntarily submits a work to a prompt claim challenge or a gift exchange challenge. The user either explicitly or implicitly agrees that their work is for challenge or prompt meme purpose, and abides by that challenge's and AO3's terms, and will be sprung from that trap by the moderator when conditions (usually, specifically, a date) are met. The moderator is a fallible human. Works can stay trapped.

In this situation, one user permitting another user to hide their work - with the risks this entails - is already part of AO3's design. It is not abuse; it is a contract of responsibility between participants and mods. Where and when, according to coders (and, I hope, the wider staff community), does this become abuse? I presume it is not abuse when a moderator says "I'm giving everyone a 2-week extension, so works will go live in April rather than March." I presume it is not abuse when the Purim Gifts exchange reveals different sets of works on different days of the festival of Purim, or when other exchanges reveal works one day at a time. I presume it is not abuse (though it is not desirable) when a moderator goes AWOL and works stay unrevealed in a collection for four years... If the issue is the idea that a work might stay indefinitely unrevealed, the "bug" fix does not fix that last example. What is the principle you are trying to code to ensure, here? That when a work is submitted to a collection by a user, this is an act of publishing, and all published works ought eventually see the light of day? That moderators may use their powers to hide works only up to a certain point, but not others? What point?

It sounds as if to coders' thinking, "When exchange challenge moderators change a collection's status to Revealed, they must reveal all works" is a Rule. Is it? Perhaps coders thought they had made it an unspoken rule by intending to code it that way, the same way that Earth has a rule that humans can't breathe water. However, that rule was not coded, and being told that we've been breaking it all this time and therefore can't now be trusted feels immensely unfair. Please let's work with what the code is, and what behaviours that has encoded in us.

You say, "We're somewhat relieved to hear it's being used with good intentions but we can't count on it remaining that way as knowledge of the bug is spread." Although I respect that your goal is to avoid harm to users, I firmly disagree with your conclusions here. If you say you knew about this before my submission, I stand corrected on that point - but so did we. Your userbase treated this as a designed function. And this is good news! Maybe we're better than you think! :) I challenge you to show that, having known about it, moderators have used it to hurt people and this is a widespread, pernicious problem worth a code fix that hurts multiple moderators acting in good faith.

It seems to me that in this discussion there are two kinds of abuse that are worth distinguishing - the kind where people are using a feature in a way it wasn't designed to be used, like Mary Poppins using an umbrella to fly, and the kind where people are taking advantage of a function to hurt others or set them up to be hurt. If you did not intend it to be possible to set a collection to be revealed, while leaving some works unrevealed, then using this function is the flying type of abuse. Agreed. However, so is the thing whereby people nominate "Group: Character/Character" in the Character slot of nominations for a tag set in an exchange. There are many kinds of not-intended-functionality abuse that I assume coders consider acceptable, because they serve a purpose that outweighs than their potential for harm (I imagine "Group: Character/Character" in the tag set character slot occasionally has negative impacts on tag wranglers). I again encourage you to consider that the use of the moderations queue to hold back works and keep them from revealing an example of this type of abuse, unintended but where harm is minor enough that it should not be a priority to fix.

You say, regarding trapping, "We note that this can be done accidentally, if the work's creator does not regularly check their works. We also note that this can be done maliciously, in the case of a moderator judging a participant's work as "not good enough"."

I respectfully submit that "if the work's creator does not regularly check their works" does not apply. Challenges are set up so that a creator can expect their work to reveal at a particular time, or during a particular period, according to other predictable factors. It's not a matter of the creator thinking "Hm, I submitted my work two months ago, should I check if it's revealed yet? Maybe I shouldn't worry now, but in two further months?" It's more a matter of "Wait, I was taking part in Exchange X, and it revealed. Why didn't my recipient get their gift?" At which point, this is a matter for mod-participant communication, or if the mod has in fact acted inappropriately, Abuse. In the circumstances in which mods would like to use the moderation queue to hold back works, I believe an affected creator has more transparency - and more cues to act - than you represent.

I also submit that if a moderator's subjective quality judgment leads to this abuse, the bug fix does not prevent it *except* that the recipient may also be involved because you want to ensure a gift notification goes out.

You say, "We want to note that, in the case of keeping the work in the moderation queue to prevent the recipient from getting a mismatched work, a malicious creator can still remove the work from the collection and directly gift it."

I take your point - this is two sides of the same coin. I'm arguing that the current moderation queue functionality is a powerful tool for moderators because it allows us to prevent works going live, but also it is not too powerful, and its potential abuse can be checked, because creators can work around it. You're arguing that if creators can work around it, it isn't really that powerful a tool. I assure you that it's valuable to me to force creators to take more effort to give something inappropriate to a recipient, giving me time to warn the recipient

I also do not want an inappropriate work to be given to a recipient with my collection's name on it. Perhaps that is mitigated by a bug fix you mention, "fixing an issue where the email notification for a rejected (and revealed) work would have the name of the collection it was rejected from." I still want to put the ball back in the creator's court if they're giving an inappropriate gift. In the case of jerks, again, this gives more time for me to mitigate the harm, and warns them that maybe they have acted inappropriately even if they didn't get my email. In the case of someone who's accidentally posted half their story that cuts off after bad HTML, or someone who posted plagiarism and was admitted to hospital, I believe that they would prefer that than that I reveal their work with the others.

Of course there will be occasions when I or other moderators make bad judgment calls. I don't think you can code for that. I think it's going to be an issue for participants and mods to sort out between themselves when it arises.

Perhaps it is more fruitful to talk about what functionality we would like to have, rather than the precise current means we achieve it. I do not expect any of this to be implemented immediately, but a) you say you're open to discussing further moderation tools and b) there is clearly a difference between what moderators think they should be able to do and what coders think they should be able to do. Let's fix that.

As far as I know, from the discussion on my Dreamwidth post (https://morbane.dreamwidth.org/41615.html), here are the things we'd like to be able to do, both as covered by the moderation queue bug, and highlighted by it.

-We do not necessarily want a means to "trap" works indefinitely. We just want a head start, or to operate on a schedule.

-When we want to hold works back from revealing for sensitive reasons, we want to protect the author (a second chance to update the work; avoiding embarrassment because the author did not finish a work and then had some kind of life crisis; preventing them from violating anonymity in another collection; a past case when an author in a very bad mental state posted a depressed, despairing diary entry about their writer's block INSTEAD of a fic, etc) or protect the recipient (spite fic; DNW fic; giving the author a further chance to update their work and therefore giving the recipient a greater likelihood of reading a finished gift; trying to avoid the author's crisis becoming the recipient's problem). I absolutely welcome further advice from Support, AD&T, and Abuse on means of handling this, but we are trying to do what we see as our jobs to facilitate good exchange experiences for everyone. We do not necessarily want to protect either participant forever, but we want further opportunity for action and mediation - to warn a participant that they're getting a sucky gift, to prevent them from getting a gift as part of a collection, etc.

-We also want to hold back works for timing reasons (Purim Gifts, Holmstice, etc).

-We would like finer control when sending out gift notifications - especially in cases of staggered works reveals.

-We would like greater transparency for authors and recipients around rejection and moderation. I sense that this is also your concern; we are with you there! One source of stress in exchanges, for me, is a difficulty getting in touch with people to explain why we are rejecting their work. A couple of people suggested on my post that on users' gift pages, and on their works pages (perhaps just visible to them in that latter case), works in a collection should show the status of rejected, approved, or awaiting approval. Or, that when a collection is revealed, creators with unrevealed works should get an automated email, like "This collection was revealed but your work was not revealed as it has been rejected".

-We would like to know what we're supposed to use Awaiting Approval and Rejected lists for.

I look forward to future solutions.

Kind regards, Morbane

Crossposts: https://morbane.livejournal.com/212577.html

Flat | Top-Level Comments Only | Expand All

From:

snickfic

I 100% agree that that is a useful behavior, and that collections should continue to have that bug/function. I can speak as an exchange moderator who has moderated a number of exchanges over the years and has not HAD to use that function but would absolutely want to be able to if the need arose.

Is there something I can do to throw in my support?

From:

morbane

Thanks! If you are willing, I'd encourage you to write to AO3 Support also asking them to preserve this function along with a simple method of using it (or better yet not change how it works right now, but I'm not hopeful of that).

AO3 collections moderation queue "bug"

Page Summary

Active Entries

Style Credit

Expand Cut Tags