Is the API serving correctly?

[eduo]

After several days without downloading almost any subs I decided to try and update all pending TV shows with subs. I loaded up SolEol and picked up all pending subtitles.

My surprise was that even though the API was letting me query results the webserver didn't actually serve these files. I kept getting HTML files saying I was over quota.

I commented this with os and my counter was reset, but I fear the error itself whatever it was wasn't fixed. Today the same happened. All the subs I'm downloading, no matter if they come from the www server or the dl server, are actually "quota exceeded" HTML files.

So, a couple of questions:

1.-Are there errors currently in OS or the OS API that are resulting in errored files?
2.-Have there been any changes to servers that could mean the API is getting confused? I had *never* got errors about exceeding quotas, and I for sure downloaded tons more subtitles than I have these two times.
3.-Is there anything I can do to avoid this?
4.-Shouldn't the API report on quotas so this can be controlled? Currently the API will happily provide results without informing the user will exceed or has exceeded his quota. There's no place or method to check the quota, either.

I'm concerned there's no way to control this and SolEol users will start hitting these errors. And I'm not even sure how to avoid them in the app or how to warn them.

Also, the subtitle download counter increases even for these subtitles, which are not being downloaded at all. It's like it addes search results instead of successful downloads.

[oss]

I am right now traveling, so no help now. Anyway, when I reset your counters, did you notice some subtitles to be counted. I'd like to know, if they was random, or they was part of search results, or...I need to check code.

I checked it last time, and didnt find any suspicious. There is 2 things going on. When user is logged in, it counts maximum downloads per userid. When it is not logged, it is limited per IP.

There have been changes in servers architecture, we are now using proxy, varnish for serving, so it means, other boxes, where is running web server and php sees other IPs. But we tried to fixed that and in tests it is working, so we can see in program which IP is accessing.

There is no reporting in API, it is possible to do, but we are focusing on OS v2.0.

When I come back, in approx 1 week I can debug more.

[scooby007]

There are loads of subtitles uploaded using soleol, but the program isn’t uploading the subtitles in the correct language field. This is a major problem and I’m guessing there are loads of subs in the database with the wrong language.

Not to mention wrong uploads from other programs, too!

Can nothing be done to make these programs more compatible? It’s all well whilst the admins go around correcting the error without user error reports being present, but what happens when admins move on or have less time to contribute on the site?

I fear the site will become rubbish as all uncategorised trash will just accumulate over a period of time, making this site go from the top five subtitle providers to being redundant.

Seriously, something needs to be done to counteract this now before it’s too late.

[eduo]

There are loads of subtitles uploaded using soleol, but the program isn’t uploading the subtitles in the correct language field. This is a major problem and I’m guessing there are loads of subs in the database with the wrong language.

Not to mention wrong uploads from other programs, too!

Can nothing be done to make these programs more compatible? It’s all well whilst the admins go around correcting the error without user error reports being present, but what happens when admins move on or have less time to contribute on the site?

I fear the site will become rubbish as all uncategorised trash will just accumulate over a period of time, making this site go from the top five subtitle providers to being redundant.

Seriously, something needs to be done to counteract this now before it’s too late.

I think the problem from SolEol is that it relied a lot on the autodetection on the site, which wasn't working for SolEol for some reason. The current version shouldn't have that problem but, of course, I can't force users to update.

The fix from SolEol subs is easy, though: Run the subs through the autodetector. It's almost always correct.

The bigger problem, to me, is all the uploads that don't even match the movie. There are TONS of those. Subtitles matched to dozens of different movies. Here the only solution is cleaning the database by removing all subtitles matched to more than one movie (multiple hashes are OK, obviously).

I was holding out on releasing a new SolEol version since all the functionality I was waiting for isn't available (Search by exact Subtitle Name/Release name, tv series support in the API, feedback on quotas) and the new API was supposed to come soon. But if OS is OK with it, I can release a version that is MANDATORY, and he can modify OS so older versions are not accepted. I can put in the release notes that the API changes and an update is mandatory. It would mean releasing a new version but I'm ok with that.

I am all in favor of fixing the situation, but I only control SolEol

[scooby007]

I am all in favor of fixing the situation, but I only control SolEol

Obviously you can only assess what you can control, but the problem isn't only SolEol, it's all the rest, too! Sub-down-loader isn't that bad at uploading from what I noticed. Even if you clean the database of all the hashes, I still think sooner or later the problem will regurgitate itself. Why? People constantly go round submitting wrong hashes to so many movie/series. Whether they have no control over this or it's done purposefully, I don't have a clue.

Point being, cleaning the database is a temporary solution. This is a good site and eventually admins won't be able to contribute the amount of time they currently do forever (they shouldn't have to); that's when users will notice all kinds of movies and shows jumbled up together, not being able to make heads or tails out of it and just leave.

I hope the new OS update can fix this issue and becomes more compatible with the programs used, but I fear that may be too late or too far away to consider as a solution. Registering to upload may make a difference as users with uploads can check what they may have uploaded (or feel compelled to check), but isn't that one of the best things about OS?

All I know is something needs to be done very soon, and we need to take these concerns very seriously. The very future of OS may depend on it!

[eduo]

I have commented this in the past, and I still think it's the way to go:

-Stop the anonymous uploads. We will get hundreds less uploads, but the wrong uploads will stop. Downloads can still be anonymous. Ask movie players to include username/pass for uploads and a pop-up message.
-Quarantine "automatic" uploads. I know they're a big source of subtitles, but they're not coded right. Subtitles seem to be being uploaded regardless of the movie they're linked to. Flagging an upload as "automatic" and automatically assigning it a low rating would be a good thing.
-Flag subtitles linked to more than one movie automatically (assigning a low ranking or even hiding them from results or the API). Keep the subtitle in the database until there are enough matches for a single movie that it can be assumed that's the correct one.
-Reduce the number of subtitles, if possible. Some movies have two dozen subs for the same release.

All of this can be implemented without modifying the API, really. I force all SolEol users to authentify, so they currently can't download nor upload without username/pass.

You are right, the site is earning a fame it shouldn't, about unreliable results and performance. Adding thousands of incorrect subs and matches can't be helping.

Eduo

[eduo]

About multiple subs. A new problem seems to have popped:

http://cl.ly/7tdL

What you see there are two subtitles with the same hash (that is, identical), the ones with the checkmarks. I've seen this happen more and more. Somehow the same subtitle is listed several times.

Also, the one with the "plus" sign is a subtitle in spanish that is incorrectly listed as english.

The results from recent tests show much better results than earlier. Has the database received a thorough cleanup?

[eduo]

As promised, the new update is almost ready to be deployed.

http://cl.ly/80Wq

Language is queried beforehand to the detectLanguage method from OS.

I'll put up some finishing touches when I come back on thursday from Portugal and release it as a mandatory update.

[scooby007]

Cheers, lad!

Does that mean it will upload in the correct language despite any problems it had from this end? If so, only if that could be done with the rest of the apps!

[eduo]

It should mean just that. The possibility for error would be greatly minimized. Users can still manually change the language to the wrong one but that would be malicious