OpenSubtitles is BLEEDING to death!

[eduo]

mm, there must be some bigger problem as it's offline again

This is what we've been trying to say. It's been a long time now that the problems can't be fixed by rebooting or restarting the servers. A change of architecture and philosophy is necessary.

[danger]

It's necessary to rewrite the application from the very beginning as it was written in the time when we had no idea about php coding and now we would take much different approaches. It's really a messed code.

I need to finish my bachelor thesis (should happen within 2 weeks) and then I proposed to os to start with rewriting the application (possibly as part of my MSc. thesis ).

[eduo]

It's necessary to rewrite the application from the very beginning as it was written in the time when we had no idea about php coding and now we would take much different approaches. It's really a messed code.

I need to finish my bachelor thesis (should happen within 2 weeks) and then I proposed to os to start with rewriting the application (possibly as part of my MSc. thesis ).

I know. os commented something not long ago. And I agree rewriting it would be a good idea, as long as the focus is not lost.

But there are plenty of things that can be done now that would help ease the load at this moment. Waiting for half a year or a whole year, being optimistic, would be suicidal. Things need to be patched up in a way they can hold until the rewrite is done (which I'm sure would take much longer than it seems).

[eduo]

Omerta:

My comments have always been on OpenSubtitles as a whole. No separation of web or API. If what I write seems overly complex then feel free to not answer to it until someone makes it clearer. I write long letters, that the way I roll. Luckily I don't force anyone to read or reply to them.

My suggestions, most of which are not made through the forum but directly, deal with trying to help making sure both the Web and the API are healthy and responsive. They have nothing to do with making the web slower, as you seem to have understood.

I'm not sure what you try to imply with "gravedigger", as I've offered both technical knowledge and hosting paid personally by me for mirroring the database and the files.

As I assume you know the traffic related to the API has now surpassed that of the website and it wouldn't surprise me if hundreds of users didn't know about the web itself. Hence care needs to be taken for both sides of the service (since it's the web side the one that makes it possible to pay for the API side).

What I won't do is stay shut when the site is down for several days ("the site" as a whole, no separation between the api and the web, as you see).

Some of us have been flagging warnings about performance for more than a year now so it's very frustrating to see our fears come to be true.

Now, unless you have proof that I don't offer actual solutions or alternatives I'd appreciate it if you didn't imply you do. If we had any option helping out instead of complaining, like we've asked for, we would.

Since I can't do anything with OpenSubtitles other than suggest and discuss then that's what I'll do, thank you very much.

Feel free to not participate in the discussion if you can't understand what the interest of the users of the site and the API are, since you seem to imply complaining for days-long outages shouldn't happen.

[eduo]

Wow boy, you did it again:DDD

Remember, I was the one who sounded the sirens first time with automatic down and uploading.

I said, buddies, careful with that! First things first!
Do we have the proper installments on site to make such experiments? Opensubtitles would remained pioneer without "bleeding itself out" to death.

You were there, you said, No, you're wrong, restrictionts are not a choice, blah-blah.

And now you're whining.

I think you're confusing me with someone else. I'm the one that first suggested limiting or cancelling anonymous uploads. You're the one that agreed to it: viewtopic.php?p=4724#4724

I later dropped off that topic because I believed there was unnecessary swearing, but that's completely besides the point.

I've also been warning about outages and offering help with mirrors and hosting for more than two years: viewtopic.php?p=3553#3553

Also, I've been lamenting the loss of the test server since mid-2007.

So, I'm not sure why you could be confusing me with someone that defended anonymous uploads or someone against restrictions. I'm all in favor of restricting access to the API, and I have a bunch of reasons.

I've been whining every time something went wrong, most of it from before you joined (officially, at least). I'm always whining. That's what you whine for: When things can be improved.

Lastly: OpenSubtitles is down. Has been down for days and intermittently for a year. Os has always been transparent about outages and limited resources so I find it baffling that you're implying I shouldn't talk about it. Precisely because "hundreds of thousand" people are looking for what's going on in OpenSubtitle is that we should talk about it. Since the blog is down as well this is the best place to talk about it.

OpenSubtitles is DOWN. It's been DOWN for days. It's a SERVER PROBLEM and the devs will fix it as soon as they can, as they've always done in the past. It's frustrating but it's better to talk about it.

[eduo]

yeah I know it's not dead, I was just being a Pessimist, cos my iPhone was ticking me off.

Anyhoo, but I was under the impression, OS did have banners and advertisements, well for the not registered users anyway?

I think your idea is pretty top notch (eduo) but wouldn't it be better to just get funding?

If funding is needed, you would't even have to contribute a huge amount.

If every user contributes taking into consideration you get thousands coming here, Literally every-day, I'm sure even $1 or £1 per year or 6 months per every user would be sufficient. Obviously you don't force it out of the user but by choice they could partake.

If OS put some kind of link on the main page where people could do the above, then atleast (hopefully) the problem could be rectified. I know i'd contribute.

This is a great site almost one of a kind, shame to see it sink into the abyss.

We're back online, luckily!

Actually, we've talked about ways to get contributions and the complicated part is that more and more the searches and downloads are being done through the API, which means banners and donation buttons wouldn't be seen.

Apps can have access to donation buttons but these aren't available yet. I have tried linking to the "Donate" page, but I'd rather be able to put a direct link to donate $1 or $5 in the app itself.

I don't think donations would sustain OpenSubtitles, though. Costs need to be brought down.

[oss]

Hi everybody,

I really apologize for what happened. I carefully read all topics here, and also all topics in forum (at least in English part:).

As Eduo wrote, I was, and still I am on holiday, luckily with my macbook. Usually I am every day on internet, checking, if website works, but I left on friday to one small island, where is no internet, and came back yesterday in night. We worked with Danger to fix server problems (web1 was down, so nothing worked, there was problem with FS, and it seems still is some kind of problem, have to ask Danger for it).

Anyway, I am listening and considering what will be the best for us (I mean for all of us). First, I'd like to put out advertisment system (openx, php, mysql), so eduo or fana, if you got 2 servers free (one for db, one for web), we can try to setup this and see how it will perform.

Also, I have to think about how I can set up mirror and what to mirror. Subtitle downloads are quite problematic, for this it needs to replicate a lot from database, also in future we will have to put some 1 line advertisement in subtitle itself (at least for API users - I will talk about that later.

Anyway, I will be back from vacation in 3 days I guess, so till now, hopefully it will work so-so. Good news is, I have plans to focus on OS development, now I set up everything locally, so it should be much better. Also I promise TV series, I will do that. Until that, no vacation!

Thanks for understanding.

[kdza09]

yeah, it's pretty annoying

- very very (very) slow

- almost everytime i get the message "500 - Internal Server Error" when loading the french page or when searching !

any improvement soon please?

[fana]

First, I'd like to put out advertisment system (openx, php, mysql), so eduo or fana, if you got 2 servers free (one for db, one for web), we can try to setup this and see how it will perform.

Sorry, if I wasn't clear about that. My offer was for the XML-RPC API and subtitle database. Not for any advertisements.

[eduo]

First, I'd like to put out advertisment system (openx, php, mysql), so eduo or fana, if you got 2 servers free (one for db, one for web), we can try to setup this and see how it will perform.

Sorry, if I wasn't clear about that. My offer was for the XML-RPC API and subtitle database. Not for any advertisements.

Fana: The current set-up of the API implies no mirroring is doable until re-designed. It would be possible, in theory, to set-up mirrors for downloading the subtitles but that wouldn't really help much with the current problems (which are related to DB, not bandwidth or file access).

[fana]

Fana: The current set-up of the API implies no mirroring is doable until re-designed. It would be possible, in theory, to set-up mirrors for downloading the subtitles but that wouldn't really help much with the current problems (which are related to DB, not bandwidth or file access).

Please help me understanding why it is not doable. Is it because the code wants to stay closed-source or the ads? Or is it something like "The IP is hardcoded in the source and no other server can use it"?

I can understand the use of advertisements if they do the paying of the running costs for infrastructure. But in my case I pay all servers from my own pocket and don't need ads. Please keep the website and ads to pay your costs.

I don't want do to mirroring for something like an "OS high availability" setup.

My idea is, that every client out there which uses the XML-RPC API should have the option to use some fallback server. Should all the clients become obsolete only because OS and its database is not available anymore?

I don't think it would be that big problem to replicate the subtitles - in terms of database or filesystem.

[eduo]

Fana: It's been said several times that the current state of OpenSubtitles is not ideal. Databases are huge, queries take a long time to complete, the files number in the thousands.

A backend and database restructuring is in the plans, seeing as how the current set-up doesn't hold. There are things that can be moved, others can't.

Now, from your comment here you don't really want to help OpenSubtitles get back on its feet, but to have a full copy of the database and API to serve it yourself. This is not what has been offered and, frankly, I don't see why it would be.

If you want to help OpenSubtitles then you can offer to either share the load or help OpenSubtitles subsist long enough for it to be rewritten in a way that facilitates distributing the workload.

If you want to "help" by setting up your own OpenSubtitles clone, and "screw OS, I pay for my own hosting" then you're clearly not what OS is looking for, with all due respect.

You're not entitled to either the sourcecode or the files. Your tone seems to imply you think you are and/or that you believe OS being closed-source is a bad thing. I'd advise you revisit this belief, as nothing has hinted this that I know of.

What some of us were offering was to *share* the workload, not to fork OpenSubtitles or to set-up standalone servers. My conversations with OS have been also in this direction trying to share the most without setting up full-blown mirrors or opening the source completely (which even if wanted is not currently in a shareable state).

So, for example, a way to share the load would be to provide mirrors for specific parts of the site that are making things difficult for the main server. Another would be hosting all the files externally. Yet another would be to have a batchjob overnight that produced mysql databases with results already produced for specific methods, and have this hosted elsewhere (so the API can redirect the methods to other places).

Other than this a lot can be done to minimize the current load of the site. One is creating static files to be served (for example all of the partial results for IMDBID can be statically created once and updated once they change, if ever, they could be concatenated in the filesystem and served without ever having to visit MySQL. Another one would be keeping directories by moviehash-bytesize with hardlinks to all subs inside them in the form md5-subtitleID-language-format so it would be VERY quick to serve them by using the filesystem tools.

But these are all suggestions to help optimize and share the load of OpenSubtitles. If there is ever the situation where OpenSubtitles goes all to hell then *that* would be the time to ask of either Danger or OS to release the source, so others can benefit from it.

[fana]

Now, from your comment here you don't really want to help OpenSubtitles get back on its feet, but to have a full copy of the database and API to serve it yourself.

From everybody who wants to do it and have the ressources. Not only by myself...

If you want to "help" by setting up your own OpenSubtitles clone, and "screw OS, I pay for my own hosting" then you're clearly not what OS is looking for, with all due respect.

I am respecting what OS have achieved until now but I think having a monopoly is bad.

You're not entitled to either the sourcecode or the files.

Yes, you're right, I am not. The copyright owner can do whatever he wants.

Your tone seems to imply you think you are and/or that you believe OS being closed-source is a bad thing.

I do.

I'd advise you revisit this belief...

I already have.

[eduo]

I believe bad monopolies are bad (I don't believe monopolies are intrinsecally bad, but then again I'm not married to any political agenda or ideology, so I'm allowed flexibility in that sense).

But OS has no monopoly over subtitles. He has made a database that uses freely available subtitle files and matches them, using a freely available and completely open hashing mechanism to 3rd party files.

He's also set-up an open API (the API is completely OPEN, you can use it freely) that uses all these open tools to search in his database for these matches.

The real value of OpenSubtitles is in that database. The API is easily replicable (I started doing it, for my internal testing, when the test server went down) and the files can be gathered from other sources (it's a PITA but it can be done).

Nothing stops you from creating a clone of OS (which would take anyone less than a couple of months to cobble together) that uses the same API.

Now, I'd rather respect the effort OS has put in this place and won't pressure him to release anything until he feels like it, and only if he feels like it. I'd personally defend his right to keep this data closed as it's that database where all his possible advantage rests, he's made everything else open.

If it were up to me I'd create a sharing protocol to distribute both the database and the files to mirrors. These mirrors in turn would all serve the API equally but they wouldn't be disconnected. I wouldn't even mind if all the data was encrypted in a way that didn't allow the mirrors to use it freely.

[oss]

Hi,

so I came back from holiday, yesterday we tried to debug what is wrong till 5.30am. Anyway, we got 3 webservers, on 1st server are some core files (NFS) and running lighttpd, on 2nd server is lighttpd, and on 3rd is lighttpd and varnish. We dont know why, but Varnish is crashing, when we enable 1st server, so now opensubtitles works only on 2 web servers, so it means it will be real slow and I think unacessible in peak hours.

We are really trying to fix this, but it is hard.

http://varnish.projects.linpro.no/ticket/506

stay tuned, it will work for sure.