500 internal server error

[oss]

eduo:

ok, we can try it. My requirements:

- we should make subdomain of opensubtitles, which can point to your server, for example help1.opensubtitles.org
- I need MYSQL, PHP, LIGHTTPD
- ssh access should be nice with root, but at least FTP.

first I would try suggestions (when you are typing in search box), if it will work stable, we can continue with some other things.

[eduo]

I was offering mostly database, disk and PHP access. The only server I can spare is my personal one, which is on a shared hosting. No Lighttpd. Since you want to replace an ajax call it shouldn't matter if it's PHP under Apache, and since it's the suggestions box the database shouldn't be enormous.

What filesize have you got for that database?

[oss]

that database is not big. Do you got memcache there ? You know, ajax suggestion box is nice, I really like it (and I hope users too), but it is quite expensive - with every typed character there is http request/db request (if not using some cache, like memcache...) and so on.

We can try that, if you want, but I think if you are on shared hosting, your hosting company will not like it

[eduo]

that database is not big. Do you got memcache there ? You know, ajax suggestion box is nice, I really like it (and I hope users too), but it is quite expensive - with every typed character there is http request/db request (if not using some cache, like memcache...) and so on.

We can try that, if you want, but I think if you are on shared hosting, your hosting company will not like it

Yeah, well. We will know when I start getting warnings from them

I know guys that handle some pretty major sites in the US. I could enquire about getting you a better deal if you let me know the details (max and actual usage) or your current contract.

Also, besides the already mentioned mirrors you can see about using Amazon S3 storage, which could reduce your traffic and bandwidth enormously ( you can check costs by using their calculator: http://calculator.s3.amazonaws.com/calc5.html ).

So far OpenSubtitles has been implemented by brute force and it's resented this now. Isn't it time to rethink it? Now that you know the real-world usage it gets there are some assumptions you can safely make.

I'd move all images and subtitles to Amazon S3. Would rethink the database ensuring the lessons learned drive the new schema and optimisations. Would change the way some things work, so data is officially updated once a day and statistics are refreshed periodically, not on-the-fly. I would also redesign the API to ensure it provides tools for the clients to avoid spamming/leeching and providing better control.

[oss]

I spoke with my friend about S3. They was hosting some images uploaded by users (something like imageshack), and Amazon just disabled account and deleted all data. Sorry, but I need something reliable, I wont study their API, code that for 2 weeks and use it for 3 months.

Currently, I am making analysis and specification of new opensubtitles, we are going to reprogram it all - also change design and so on. So all those things is for new OS.

Nowadyas I add some small things:
- in user profile user can list requested subtitles
- in subtitle detail page is fetched last 10 subtitles for same language...

Now I will try to focus of better handling duplicates subtitles and some old bugfixes.

[eduo]

I spoke with my friend about S3. They was hosting some images uploaded by users (something like imageshack), and Amazon just disabled account and deleted all data. Sorry, but I need something reliable, I wont study their API, code that for 2 weeks and use it for 3 months.

Currently, I am making analysis and specification of new opensubtitles, we are going to reprogram it all - also change design and so on. So all those things is for new OS.

Nowadyas I add some small things:
- in user profile user can list requested subtitles
- in subtitle detail page is fetched last 10 subtitles for same language...

Now I will try to focus of better handling duplicates subtitles and some old bugfixes.

I don't want to imply your friend hasn't got this right, but S3 is being used by pretty large sites, like SmugMug, that host pictures of all types. It also hosts every image in Twitter, which ranges from copyright infringements to porn.

Also, they wouldn't close and delete an files without warning, something they clarify in their FAQ. They would, at worst, give a warning to retrieve all files.

There is no API to learn how to use. There are tons of clients to upload files to S3, including command-line and PHP. And the URL is static. You can start using S3 in less than a day and if you keep a local copy of every file you can stop using it at any time.

If you're worried about leeching then that is something else, but not really related to S3. It's a pretty simple change that would have an immediate benefit, even if it worked for just a few months in the worst-case.

Anyway, you know better. I'd advise against the Second-System Syndrome. Planning ro re-design OS from scratch might be too much to do in one go.

[Sorku]

that database is not big. Do you got memcache there ? You know, ajax suggestion box is nice, I really like it (and I hope users too), but it is quite expensive - with every typed character there is http request/db request (if not using some cache, like memcache...) and so on.

We can try that, if you want, but I think if you are on shared hosting, your hosting company will not like it

I haven't looked into how os handles the ajax suggestion box, but I hope that you're aware that it's very easy to put a delay timer on the http requests in javascript, so it only sends one request x seconds after the user has stopped typing.

Another way to handle the ajax search might be to use cached serialized objects of all titles and filter the input data in php instead of mysql?

[eduo]

that database is not big. Do you got memcache there ? You know, ajax suggestion box is nice, I really like it (and I hope users too), but it is quite expensive - with every typed character there is http request/db request (if not using some cache, like memcache...) and so on.

We can try that, if you want, but I think if you are on shared hosting, your hosting company will not like it

I haven't looked into how os handles the ajax suggestion box, but I hope that you're aware that it's very easy to put a delay timer on the http requests in javascript, so it only sends one request x seconds after the user has stopped typing.

Another way to handle the ajax search might be to use cached serialized objects of all titles and filter the input data in php instead of mysql?

Yup, pretty basic. Caching should happen at server level through PHP and at client level at JS level (to avoid people deleting half a word to create a query for each stroke)

[oss]

thanks for suggestions, I did not have there "frequency", so I've added it - more reading here http://wiki.github.com/madrobby/scripta ... ocompleter

also for those values, I am already storing them to memcached, so it shouldnt be issue.

We got problems with TCP overflow to DB server, thats why we are getting these errors, more reading here:

https://forums.freebsd.org/showthread.php?p=26250

Now I turned off connections to mysql slave server, we will see how it will perform now - but still issue is not solved, just workaround...looking for cure.

[oss]

Hello all,

we know few last days was terrible - you can not access website you can not use XMLRPC. I've been asking around of forums how to fix our server issues, I got some advices, I implemented them with DanGer, it helps only little bit. Then I remembered #lighttpd channel on irc, and asking around.

I found a great guy, who helps us a lot, so hopefully OS will work much better.

Now there are still some issues, but believe us, we are working on fixing these stuff (last time I went sleep 6.00am...)

Please be patient and keep supporting us.

Thank you all.

[eduo]

Hi there

What ended up being a problem? What was this guy able to find?

[eduo]

Hello? Any news at all? Even the trac is down...

[oss]

hi yes, yesterday was lucky day for me, it seems my friend fixed some problems. He changed some settings of webservers (now is lighttpd on one server, on other are php-cgi), also tune up some kernel things, for example this is db server:

Code: Select all

kern.ipc.maxsockets="131072" vm.pmap.pg_ps_enabled=1 vm.kmem_size=1G kern.ipc.somaxconn=16384 kern.ipc.nmbclusters=32768 security.bsd.see_other_uids=0 security.bsd.see_other_gids=0 net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.icmp.icmplim=150 net.inet.icmp.drop_redirect=1 security.bsd.unprivileged_read_msgbuf=0 kern.maxfiles=131072 kern.maxfilesperproc=104856 kern.threads.max_threads_per_proc=4096 net.inet.tcp.keepinit=1000 net.inet.ip.portrange.last=65535 net.inet.ip.portrange.first=10000

for now I turned off XMLRPC for unauthenticated users, I will do some more optimalizations (using more memcached, if possible). I hope better times are coming.

If you find out something wrong, please report it.

Thanks to everybody.

[eduo]

hi yes, yesterday was lucky day for me, it seems my friend fixed some problems. He changed some settings of webservers (now is lighttpd on one server, on other are php-cgi), also tune up some kernel things, for example this is db server:

Code: Select all

kern.ipc.maxsockets="131072" vm.pmap.pg_ps_enabled=1 vm.kmem_size=1G kern.ipc.somaxconn=16384 kern.ipc.nmbclusters=32768 security.bsd.see_other_uids=0 security.bsd.see_other_gids=0 net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.icmp.icmplim=150 net.inet.icmp.drop_redirect=1 security.bsd.unprivileged_read_msgbuf=0 kern.maxfiles=131072 kern.maxfilesperproc=104856 kern.threads.max_threads_per_proc=4096 net.inet.tcp.keepinit=1000 net.inet.ip.portrange.last=65535 net.inet.ip.portrange.first=10000

for now I turned off XMLRPC for unauthenticated users, I will do some more optimalizations (using more memcached, if possible). I hope better times are coming.

If you find out something wrong, please report it.

Thanks to everybody.

Trac is still down with varnish errors. I assume the API is in the same situation.

[danger]

it should be much better now.