How to deal with duplicate subs

[oss]

TV series - I already have programmed that (but dev server gone:), it should not be a big deal, but not now, it is in my todo list.

for updating subtitles - I can code that, but we have to think about that. What about this progress:

1. you upload 2CD subtitles (logged in)
2. you want change 1CD or 2CD, so you are logged in web, click upload on 1CD or 2CD, then it will ask ONLY for subtitle file, and thats it - old file will be deleted and new one will appear.

Something like that ?

[oss]

ok, question is, if we dont need some track of those subtitles. My solution is really simple - you dont know those subtitles was updated and that should be problem (you can write in in user comments, but...), but more complicated (tracking old subtitles) solution takes more time to program it.

Now I am thinking - I just put a upload button next to each subtitle file, and thats it, nothing more - you just replace that file.

[oss]

for filenames - I am tracking filenames (they are not visible on page, but they can be, seen sort desc), I think default filename should be most uploaded filename - and that should be done automatically. First I will add this feature to detail of subtitles, so you will see, if it is good idea to replace default filename with most TryUpload() filename...

[oss]

yes, I understand. another good idea should be: if the filename is not changed by admin, use most used filename based on seen_number

[oss]

hello,

I started training muay thai again, so not so much time for OS, also I bought Mac Book (black), so I have fun with it.

Anyway, every day I work also on opensubtitles, as you can see. I understand, you, as polish admin have too much work and you need better admin menu.

If you can, you should describe every point of your message as much as possible, so I can start coding that. For example I can imagine multidelete (checkboxes, or ajax)...

for limiting subtitles for 3 or 5 releases. Now system is (should be) quite good:

if there is no similar subtitle (same imdb, same language, almost same timestamps), upload it. If you will delete that subtitle, also moviehash is lost, so people cannot download that subtitle. In future I think, more and more people will access this site using xmlrpc, so there is important to have those hashes.

[oss]

hi,

now upload filter for xmlrpc for anonymous users is set like this:

Code: Select all

if(anonymous_uploader) { if(similar subtitles based on timestamps found for same language) { save moviehash dont upload } else { upload } } if(anonymous_uploader) { if(count(idmovie,sublanguage,moviehash,moviebytesize) >= 1 ) { dont save subtitle, save only subtitle hash: } else { upload subtitle }

[everthonvs]

(I started this message as a reply to an admin complaining about some duplicate, and as the reply gets bigger I decided to post it here to enable discussion and further viewing).

Sometimes I downloaded some subtitle directly from legendas.tv* and uploaded here WITHOUT any change. And very often situations like these happened: I upload it latter than expected, so someone else download it there, edit the original to add some of his "credits" and upload it here first. And when I upload the *original* translation here, the hash differ so opensubtitles accept it.
(* legendas.tv is the current Brazilian hot site for the major subtitle teams, and yes, I invited some of the teams to join the opensubtitles

So, I understand the huge problem you guys have with duplicates. This common situation lead us to some important issue: the "hash" algorithm maybe isn't the best way to deal with duplicates. Maybe the OS admins should think about applying a diff on the subtitles for the same release and maybe ask the user for "are you really sure it isn't a duplicate?" and show the diff lines or something like that.

Sometimes I have to correct only 3 or 5 misspelled words and bad-translated ones and how neat it will be if I could *update* the subtitle instead of re-uploading a new one?

Diff will be nice, but a better feature to implement is a way to collaborate and manage version controlling of the subtitle production, and THAT will be very welcome. I'm thinking about something like the Rosetta web-translation on Launchpad.

Another reason why the file-size "hash" isn't so good: When I re-sync some subtitle for another release, I *must* change some char or add a new line break because the "hash" is based on simple file-size: the re-sync'd and the subtitle for another release are different but the hash tells me they are not. And it gets worse: sometimes when I try to upload a subtitle for some TV show, the "hash" tells me that I can't because it is supposedly the same subtitle for some unrelated obscure 60's movie.

I like to say thank you all for the hard work and dedication on building and maintaining the so important initiative that Opensubtitles are. I'm a registered user since 2007 and I'm uploading every subtitle I made, re-sync or download from another source for Brazilian Portuguese. And I have to tell you: Opensubtitles was *very* important on the begin of 2009, when police rages shut down the major Brazilian subtitle sites. And after that, Opensubtitle grow big on the Brazilian hearts and collaborators come more and more everyday!

BTW, I use Subdownloader to easily upload the subtitles I get from another sources.

[eduo]

Actually, the hash for subtitles is MD5, not related to filesize at all.

Duplicates have been a problem with OpenSubtitles and the subtitles scene for a long time. I've mentioned before what I believe needs to happen before it can be fixed: Have only official sources for subtitles.

It's what finally put the other "scenes" into order anyway, having release groups.

Now, these official sources to me can be anything, but I'd prefer them to be sites based on the wikititles codebase, because they can track creators, editors, translators and versions. They ensure no duplicates with just the credits changed exist. They can also demonstrate in a dispute what the original subtitle is. And they add a layer of trust to the whole thing which is currently missing.

No site does this right now. There's a disconnect between subtitle creators and editors and subtitle repositories, and there's lots of friction because the former consider the latter to be no more than leeches (which, depending on the way you interpret the word, they actually are, as are VTV and EZTV for the video scene).

I tried contacting the Addic7ed guys for this without much luck, because of said "bad image". But this can still be done. The code for the wiki can be provided to have a working API and a deal can be agreed with these sites:

-OpenSubtitles serves the files from the wikis (helping them with the bandwidth)
-The wikis control the subs and the versioning. Only one sub per wiki per release per language. If a new version appears, it overwrites the old one.
-OpenSubtitles allows editing of subtitles by redirecting the user to the proper wiki.

I can only see everyone winning with this proposal. They get more exposure and less bandwidth charges. OpenSubtitles gains credibility and reliability and the users, clearly, benefit from increased quality.

[ckfeng]

I agree TMP format really sucks, it is not supported in many players

I can change algo, so it will NOT search for similar subs in TMP subtitles, I think that should do a job... or searching in all formats and when it found similar subs in TMP format disable TMP and upload other format. But I like first option first...