Bug fixes and new features

Hello, everyone! We think you lot might be interested in a few updates.

We’ve deployed some bug fixes lately, including the following:

  • deviantART changed their download links, presumably to stop automated scrapers. Our automated scraper is now fixed, since they won’t fix their oEmbed API to provide links to full images, so you can once again drop dA deviation URIs in to get the image along with an appropriate artist tag and source URI set
  • The changelog is back!
  • Notifications on deleted/dupe-merged images won’t linger any more, and the displayed count will always be accurate/consistent
  • Minor bug fixes to the forums, including a bug where posts might end up in the wrong topic if the title was the same across multiple forums
  • Fixes to the new duplicate reporting system

On the new features front - well, nothing much for now, but we’ve redesigned the lists page - which was intended to give a nice overview of site activity - into an actual activity page. This page has the latest images, comments, and forum posts/live streams, as well as the usual recent/all-time top images.

We’re also fiddling a bit with the deduplication system to let us manage reports more readily and to allow uploaders more visibility about reports occurring. Our new LSH system ran into some kinks which we’re still working out but the slightly tweaked existing perceptual dedupe system is performing better in any case.

Love,
Clover, K_A and the rest of the team

Duplicates, duplicates, duplicates, duplicates

This is a post about dupes.

Roughly 10% of the images that people upload to the site we actually already have. We A) don’t want to store those, and B) don’t want to split up discussions on images around multiple versions, and so on.

Dupes are bad. We’ve been better than many boards at stopping duplication from taking over - that 10% number comes from the number that we’ve gotten merged, and we don’t miss many.

However, what we had has not been quite good enough. People intentionally try and work around the deduplicator to get images to appear on the front page of the site more often, for instance. Some people just want to steal upload credit. Usually people made a report to us with something like “Dupe of »1234” and we’d sort it, but it is a lot of effort and time taken away from the mods.

So we’ve obviously been trying to figure out a better way to do things, and we’re rolling out the first part of that now. That is duplicate reports. You can click “Report a duplicate” on any image and see automated and manually submitted reports. If you uploaded an image, you’ll be able to accept or reject reports - accepting the report merges your upload, while rejecting it makes it clear you don’t think that the image is a duplicate. This historically had to be done by an administrator but now we have more auditability of the system, we’re reopening it to users as a trial.

More interesting, though, is that we’ve added a new perceptual deduplication system. Our old system compared about 6 parameters about an image. The new one compares 7203. It’s a bit more robust. We’re using some fairly fancy stuff (including locality sensitive hashing) and it is very experimental. We’re looking forward to seeing how it performs, and fully expect to have to tweak it a few times before we get the right balance between false positives and false negatives. We’ll be running it in parallel with our old one for now.

Deduplication checks now run asynchronous to uploads, so once you hit upload you may not immediately see any dupes. We’re working on improving this workflow through a new upload process, but this isn’t done yet.

As ever, feedback to the forums please!

Duplicates, duplicates, duplicates, duplicates

This is a post about dupes.

Roughly 10% of the images that people upload to the site we actually already have. We A) don’t want to store those, and B) don’t want to split up discussions on images around multiple versions, and so on.

Dupes are bad. We’ve been better than many boards at stopping duplication from taking over - that 10% number comes from the number that we’ve gotten merged, and we don’t miss many.

However, what we had has not been quite good enough. People intentionally try and work around the deduplicator to get images to appear on the front page of the site more often, for instance. Some people just want to steal upload credit. Usually people made a report to us with something like “Dupe of »1234” and we’d sort it, but it is a lot of effort and time taken away from the mods.

So we’ve obviously been trying to figure out a better way to do things, and we’re rolling out the first part of that now. That is duplicate reports. You can click “Report a duplicate” on any image and see automated and manually submitted reports. If you uploaded an image, you’ll be able to accept or reject reports - accepting the report merges your upload, while rejecting it makes it clear you don’t think that the image is a duplicate. This historically had to be done by an administrator but now we have more auditability of the system, we’re reopening it to users as a trial.

More interesting, though, is that we’ve added a new perceptual deduplication system. Our old system compared about 6 parameters about an image. The new one compares 7203. It’s a bit more robust. We’re using some fairly fancy stuff (including locality sensitive hashing) and it is very experimental. We’re looking forward to seeing how it performs, and fully expect to have to tweak it a few times before we get the right balance between false positives and false negatives. We’ll be running it in parallel with our old one for now.

Deduplication checks now run asynchronous to uploads, so once you hit upload you may not immediately see any dupes. We’re working on improving this workflow through a new upload process, but this isn’t done yet.

As ever, feedback to the forums please!

Livestreams

So the observant amongst you will have noticed we’ve added a new button on the site - Streams.

This takes you to a page where we’ve collected a user-contributed list of Livestream accounts used by pony artists, listing many of your favourite artists and letting you easily click through to live streams to see them working their magic. You can also go from the livestream page straight to their artist tag on Derpibooru to browse their artworks. There’s a toggle to view NSFW streams, too.

You can also subscribe to livestreams - this sends you a notification when your favourite artists begin to stream, which is cleared when the stream closes.

We hope people will enjoy this feature - we’ve been wanting to add it for a little while and it was a quick thing for us to whip up. You can send feedback to us via the forums - there’s a thread in the site forum.

If you’re an artist who streams or would like to list an artist you know, send The Smiling Pony a private message - the link is on the bottom of the livestreams page.

Love,
Clover

Top