Page 1 of 7

[NEW] REST search, JSON results

Posted: Mon Mar 26, 2018 10:52 am
by oss
Hi guys,

More and more clients are connecting to opensubtitles API using XML-RPC which is difficult to cache. We are proud to present you REST search. Idea is simple - client make URL, request it and we cache results on CDN (in Cloudflare), that means, responses are faster and our servers can take care of something else.

Make sure you provide your XML-RPC useragent (API) in http header. (#curl -A 'registeredUA' or <?php curl_setopt($curl, CURLOPT_USERAGENT, $agent); )

OK, how it looks?

Code: Select all

https://rest.opensubtitles.org/search/episode-20/imdbid-4145054/moviebytesize-750005572/moviehash-319b23c54e9cf314/season-2/sublanguageid-eng
to create request you need base URL, which is https://rest.opensubtitles.org/search and then search parameters ordered by name, otherwise you will be redirected (please sort them on client side!). Make sure url is in _lowercase_, otherwise redirect. Valid parameters are:

Code: Select all

episode (number) imdbid (always format it as sprintf("%07d", $imdb) - when using imdb you can add /tags-hdtv for example. moviebytesize (number) moviehash (should be always 16 character, must be together with moviebytesize) query (use url_encode, make sure " " is converted to "%20") season (number) sublanguageid (if ommited, all languages are returned) tag (use url_encode, make sure " " is converted to "%20")
Valid URLs (examples, try yourself to combine):

Code: Select all

curl -A 'TemporaryUserAgent' https://rest.opensubtitles.org/search/moviebytesize-750005572/moviehash-319b23c54e9cf314 curl -A 'TemporaryUserAgent' https://rest.opensubtitles.org/search/episode-20/imdbid-4145054/moviebytesize-750005572/moviehash-319b23c54e9cf314/season-2/sublanguageid-eng curl -A 'TemporaryUserAgent' https://rest.opensubtitles.org/search/episode-11/imdbid-4145054/season-1/tags-web-dl curl -A 'TemporaryUserAgent' https://rest.opensubtitles.org/search/query-matrix%20reloaded/sublanguageid-eng curl -A 'TemporaryUserAgent' https://rest.opensubtitles.org/search/episode-8/sublanguageid-eng/tag-heroess01e08
NEW: for clients unable to change default User Agent it is possible now to set up custom HTTP header 'X-User-Agent':

Code: Select all

curl --header "X-User-Agent: TemporaryUserAgent" https://rest.opensubtitles.org/search/query-matrix%20reloaded/sublanguageid-cze

INVALID URLs:

Code: Select all

# duplicate parameters are not permitted https://rest.opensubtitles.org/search/imdbid-4145054/imdbid-1234567 # duplicate values per parameter are not permitted https://rest.opensubtitles.org/search/imdbid-4145054/sublanguageid-eng,cze
Results are limited to 500 items. This REST method (search) is public and doesn't need any key or any special user-agent. Results are sorted by SCORE, so which are first in list should be better.
Always check for 200 response http code.
Example:

Code: Select all

# https://rest.opensubtitles.org/search/episode-1/imdbid-4145054/season-1/sublanguageid-cze < HTTP/2 200 < date: Mon, 26 Mar 2018 08:48:54 GMT < content-type: application/json < set-cookie: __cfduid=debccacb67b21029477d3d105bd3943131522054134; expires=Tue, 26-Mar-19 08:48:54 GMT; path=/; domain=.opensubtitles.org; HttpOnly < cache-control: public, max-age=43200 < last-modified: Mon, 26 Mar 2018 10:47:00 +0200 < access-control-allow-origin: * < access-control-allow-headers: ETag, X-RateLimit-Remaining, Content-Type, Last-Modified < x-robots-tag: noindex, nofollow < vary: Accept-Encoding < x-cache-backend: web3 < x-cache: MISS < x-ratelimit-remaining: 40 < x-via: fw2.int.opensubtitles.org < cf-cache-status: HIT < expires: Mon, 26 Mar 2018 20:48:54 GMT < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" < server: cloudflare < cf-ray: 40185c628ecb6f7e-SIN [{"MatchedBy":"imdbid","IDSubMovieFile":"0","MovieHash":"0","MovieByteSize":"0","MovieTimeMS":"0","IDSubtitleFile":"1955043364","SubFileName":"Shadowhunters.S01E01.HDTV.x264-KILLERS.srt","SubActualCD":"1","SubSize":"42803","SubHash":"66fea5461f173f689c3675c372cd055a","SubLastTS":"00:38:57","SubTSGroup":"1","IDSubtitle":"6461000","UserID":"0","SubLanguageID":"cze","SubFormat":"srt","SubSumCD":"1","SubAuthorComment":"","SubAddDate":"2016-01-13 23:19:41","SubBad":"0","SubRating":"0.0","SubSumVotes":"0","SubDownloadsCnt":"1332","MovieReleaseName":" Shadowhunters.S01E01.HDTV.x264-KILLERS","MovieFPS":"0.000","IDMovie":"406174","IDMovieImdb":"4601794","MovieName":"\"Shadowhunters: The Mortal Instruments\" The Mortal Cup","MovieNameEng":null,"MovieYear":"2016","MovieImdbRating":"7.6","SubFeatured":"0","UserNickName":"","SubTranslator":"","ISO639":"cs","LanguageName":"Czech","SubComments":"0","SubHearingImpaired":"0","UserRank":"","SeriesSeason":"1","SeriesEpisode":"1","MovieKind":"episode","SubHD":"0","SeriesIMDBParent":"4145054","SubEncoding":"UTF-8","SubAutoTranslation":"0","SubForeignPartsOnly":"0","SubFromTrusted":"0","SubTSGroupHash":"a5985421faabcab605e61243794a4d7f","SubDownloadLink":"https:\/\/dl.opensubtitles.org\/en\/download\/src-api\/vrf-19a20c51\/filead\/1955043364.gz","ZipDownloadLink":"https:\/\/dl.opensubtitles.org\/en\/download\/src-api\/vrf-f52d0baa\/subad\/6461000","SubtitlesLink":"http:\/\/www.opensubtitles.org\/en\/subtitles\/6461000\/shadowhunters-the-mortal-instruments-the-mortal-cup-cs","QueryNumber":"0","QueryParameters":{"episode":1,"season":1,"imdbid":"4145054","sublanguageid":"cze"},"Score":10.01332},{"MatchedBy":"imdbid","IDSubMovieFile":"0","MovieHash":"0","MovieByteSize":"0","MovieTimeMS":"0","IDSubtitleFile":"1955231797","SubFileName":"Shadowhunters S01E02 The Descent Into Hell Isn't Easy.srt","SubActualCD":"1","SubSize":"51130","SubHash":"5ec1754f14ed36e087a5c799815fadc9","SubLastTS":"00:40:41","SubTSGroup":"3","IDSubtitle":"6651521","UserID":"0","SubLanguageID":"cze","SubFormat":"srt","SubSumCD":"1","SubAuthorComment":"","SubAddDate":"2016-06-08 15:54:26","SubBad":"0","SubRating":"0.0","SubSumVotes":"0","SubDownloadsCnt":"170","MovieReleaseName":" Shadowhunters S01E02 The Descent Into Hell Isn't Easy","MovieFPS":"25.000","IDMovie":"406174","IDMovieImdb":"4601794","MovieName":"\"Shadowhunters: The Mortal Instruments\" The Mortal Cup","MovieNameEng":null,"MovieYear":"2016","MovieImdbRating":"7.6","SubFeatured":"0","UserNickName":"","SubTranslator":"","ISO639":"cs","LanguageName":"Czech","SubComments":"0","SubHearingImpaired":"0","UserRank":"","SeriesSeason":"1","SeriesEpisode":"1","MovieKind":"episode","SubHD":"0","SeriesIMDBParent":"4145054","SubEncoding":"CP1250","SubAutoTranslation":"0","SubForeignPartsOnly":"0","SubFromTrusted":"0","SubTSGroupHash":"a201acec74b8fdf5f64130bda9a7c5c7","SubDownloadLink":"https:\/\/dl.opensubtitles.org\/en\/download\/src-api\/vrf-19b60c5a\/filead\/1955231797.gz","ZipDownloadLink":"https:\/\/dl.opensubtitles.org\/en\/download\/src-api\/vrf-f5480bb3\/subad\/6651521","SubtitlesLink":"http:\/\/www.opensubtitles.org\/en\/subtitles\/6651521\/shadowhunters-the-mortal-instruments-the-mortal-cup-cs","QueryNumber":"0","QueryParameters":{"episode":1,"season":1,"imdbid":"4145054","sublanguageid":"cze"},"Score":10.0017},{"MatchedBy":"imdbid","IDSubMovieFile":"0","MovieHash":"0","MovieByteSize":"0","MovieTimeMS":"0","IDSubtitleFile":"1955209010","SubFileName":"Shadowhunters - 01x01 - The Mortal Cup (Pilot).KILLERS.English.HI.C.orig.Addic7ed.com.srt","SubActualCD":"1","SubSize":"41037","SubHash":"b67051a0013745caf50c0706c2a954ae","SubLastTS":"00:38:58","SubTSGroup":"1","IDSubtitle":"6628618","UserID":"0","SubLanguageID":"cze","SubFormat":"srt","SubSumCD":"1","SubAuthorComment":"","SubAddDate":"2016-05-17 13:33:06","SubBad":"0","SubRating":"0.0","SubSumVotes":"0","SubDownloadsCnt":"79","MovieReleaseName":" Shadowhunters - 01x01 - The Mortal Cup (Pilot).KILLERS.English.HI.C.orig.Addic7ed.com","MovieFPS":"23.976","IDMovie":"406174","IDMovieImdb":"4601794","MovieName":"\"Shadowhunters: The Mortal Instruments\" The Mortal Cup","MovieNameEng":"","MovieYear":"2016","MovieImdbRating":"7.6","SubFeatured":"0","UserNickName":"","SubTranslator":"","ISO639":"cs","LanguageName":"Czech","SubComments":"0","SubHearingImpaired":"0","UserRank":"","SeriesSeason":"1","SeriesEpisode":"1","MovieKind":"episode","SubHD":"0","SeriesIMDBParent":"4145054","SubEncoding":"CP1250","SubAutoTranslation":"0","SubForeignPartsOnly":"0","SubFromTrusted":"0","SubTSGroupHash":"a5985421faabcab605e61243794a4d7f","SubDownloadLink":"https:\/\/dl.opensubtitles.org\/en\/download\/src-api\/vrf-199b0c49\/filead\/1955209010.gz","ZipDownloadLink":"https:\/\/dl.opensubtitles.org\/en\/download\/src-api\/vrf-f55d0bbe\/subad\/6628618","SubtitlesLink":"http:\/\/www.opensubtitles.org\/en\/subtitles\/6628618\/shadowhunters-the-mortal-instruments-the-mortal-cup-cs","QueryNumber":"0","QueryParameters":{"episode":1,"season":1,"imdbid":"4145054","sublanguageid":"cze"},"Score":10.00079},{"MatchedBy":"imdbid","IDSubMovieFile":"0","MovieHash":"0","MovieByteSize":"0","MovieTimeMS":"0","IDSubtitleFile":"1955105830","SubFileName":"Shadowhunters.S01E01.1080p.WEB-DL.DD5.1.H264-RARBG.srt","SubActualCD":"1","SubSize":"41135","SubHash":"2a012782be0d93f31c4053669d6c2f76","SubLastTS":"00:39:05","SubTSGroup":"5","IDSubtitle":"6524935","UserID":"0","SubLanguageID":"cze","SubFormat":"srt","SubSumCD":"1","SubAuthorComment":"","SubAddDate":"2016-02-25 21:36:46","SubBad":"0","SubRating":"0.0","SubSumVotes":"0","SubDownloadsCnt":"74","MovieReleaseName":" Shadowhunters.S01E01.1080p.WEB-DL.DD5.1.H264-RARBG","MovieFPS":"23.976","IDMovie":"406174","IDMovieImdb":"4601794","MovieName":"\"Shadowhunters: The Mortal Instruments\" The Mortal Cup","MovieNameEng":"","MovieYear":"2016","MovieImdbRating":"7.6","SubFeatured":"0","UserNickName":"","SubTranslator":"","ISO639":"cs","LanguageName":"Czech","SubComments":"0","SubHearingImpaired":"0","UserRank":"","SeriesSeason":"1","SeriesEpisode":"1","MovieKind":"episode","SubHD":"1","SeriesIMDBParent":"4145054","SubEncoding":"CP1250","SubAutoTranslation":"0","SubForeignPartsOnly":"0","SubFromTrusted":"0","SubTSGroupHash":"ad6e0f888438099ed83c2d1a335588bd","SubDownloadLink":"https:\/\/dl.opensubtitles.org\/en\/download\/src-api\/vrf-19a10c4e\/filead\/1955105830.gz","ZipDownloadLink":"https:\/\/dl.opensubtitles.org\/en\/download\/src-api\/vrf-f5510bbb\/subad\/6524935","SubtitlesLink":"http:\/\/www.opensubtitles.org\/en\/subtitles\/6524935\/shadowhunters-the-mortal-instruments-the-mortal-cup-cs","QueryNumber":"0","QueryParameters":{"episode":1,"season":1,"imdbid":"4145054","sublanguageid":"cze"},"Score":10.00074}]
If you will find out this useful, let us know.

More REST methods to play with:

Code: Select all

curl -A 'TemporaryUserAgent' http://rest.opensubtitles.org/suggest/query-a/sublanguageid-eng curl -A 'TemporaryUserAgent' http://rest.opensubtitles.org/getsublanguages/language-en curl -vvv -d 'username=your_username&password=your_password' -X POST -A 'TemporaryUserAgent v1.2' https://rest.opensubtitles.org/auth curl -vvv --user username:password -A 'TemporaryUserAgent v1.2' https://rest.opensubtitles.org/auth
Example passing SID parameter for download URL:

Code: Select all

http://dl.opensubtitles.org/en/download/src-api/vrf-19d50c5e/sid-{YOUR_SID_HERE}/filead/1951976249.gz
If you want to download subtitles as logged in user, either use http basic auth, or add sid parameter into download link.

Re: [NEW] REST search, JSON results

Posted: Mon Mar 26, 2018 8:38 pm
by eduo
Nice.

What happens if only bytesize and hash are provided but not IMDBid? Or only IMDBid?

I understand that if I have 50 files (2 tv seasons) I must make 50 queries for each subtitle language. What are the limits (based on IP, I guess) for this method?

What agent should we use? Not sure if you're keeping statistics.

Re: [NEW] REST search, JSON results

Posted: Tue Mar 27, 2018 3:50 am
by oss
Hi Eduo, long time...

to answer your questions - you can try. Of course you can query just by imdbid:
https://rest.opensubtitles.org/search/i ... uageid-cze

or just moviehash and moviebytesize:
https://rest.opensubtitles.org/search/m ... c54e9cf314

if you have 50 files, you have to make 50 queries (by moviehash for example), yes. If you want 1 language not a problem, if you want all languages not a problem, but if you want 2 languages, you have to make 2 queries per file - it is done for sake of caching on CF (better hit ratio), but more requests...I know.

there is not limits, no UA needed just speed limit: 40 queries per IP per 10 seconds. We not making any stats for now.

This should help to avoid LogIn() in XMLRPC and caching better in CF.

But you are right, maybe we should request setting HTTP UA to actually know what is going on.

EDIT: I added UA checking, it makes sense.

Re: [NEW] REST search, JSON results

Posted: Tue Mar 27, 2018 2:28 pm
by noembryo
Can you please post an example of an http header with a fake XML-RPC useragent in it?
Do you mean instead of "Mozilla/5.0 (Windows NT 5.1; rv:14.0) Gecko/20100101 Firefox/24.0.1" or whatever, I should use my app's user agent in the http request?

The rest of the calls look really straightforward (even for me ;o)

Re: [NEW] REST search, JSON results

Posted: Tue Mar 27, 2018 3:47 pm
by oss
hi, yes instead of standard HTTP user agent you must use the user agent, which is registered for xml-rpc. I post example using "curl" command, in every language you define different user agent, for sure you will find it.

Re: [NEW] REST search, JSON results

Posted: Thu Mar 29, 2018 8:03 pm
by noembryo
I tried the rest API today and it looks very nice.
There are some issues though.
One has to do with the subtitles returned.
Alternating between the XML-RPC and the JSON method does not return the same results for the same file.
JSON returns less some of the times.
With XML-RPC I use only the hash and the size (and the language).
With the JSON I also add the title as query, thinking that it might get me more results.
I tried with the title omitted and got the same results some of the times.
This is strange.

The other thing has to do with the limit of queries (40).
Almost all my friends that use Subber always search for at least 2 languages. English and their native one.
If they drop a Series folder to search, the number of files can easily get more than 20.
One season of a series can be more than 20.
Some series can get more than 200.
It might be more helpful to be able to send an array of languageIDs with the REST and get only these returns.
As it is now, I get all the languages and then I filter the ones I don't need.
This works but is a little wasteful.
Perhaps the limit is a tad low...

Re: [NEW] REST search, JSON results

Posted: Fri Mar 30, 2018 11:01 pm
by darkvinill
Should I use JSON instead XML-RPC ?

Re: [NEW] REST search, JSON results

Posted: Tue Apr 03, 2018 2:41 pm
by kioo123
JSON API means you can issue a JSON request to server just like other JSON requests.
Because the JSON API is PUBLIC that means you will not need LOGIN credentials, and that is one request less to the OPENSUBTITILES servers.
Also, since the requests are cached, there is faster response too. Keep in mind that your USERAGENT must be registered and approved in OPENSUBTITLES or you will get a response "http useragent is not valid"
To make a simple test to the API, use a REST Client such as in Firefox you can install the AddOn RESTED client:
Open it, then enter the following:

URL: https://rest.opensubtitles.org/search/m ... c54e9cf314

Headers -> Add a New Header and on name field enter: User-Agent and on it value field enter "TemporaryUserAgent" don't forget the quotes.
Remember "TemporaryUserAgent" is for Development only and it has limitation; hence don't use it in production and instead use yours.

Click Submit, and Voilah! Your response will be there in nicely formatted JSON.

Hope this helps

Re: [NEW] REST search, JSON results

Posted: Tue Apr 03, 2018 5:22 pm
by kioo123
Hi OpenSubttitles,

I was trying to use the new API in my application written in QT, but unfortunately QT doesn't allow change of User-Agent in the request header. It always defaults to Mozilla/5.0 as User-Agent.
Is it possible to use a custom request header name instead in order to pass this restriction?

Re: [NEW] REST search, JSON results

Posted: Wed Apr 04, 2018 12:05 am
by oss
Hi,

for QT set UA, quick google shows this: https://wiki.qt.io/How_to_set_user_agen ... pplication

it is not solution for your problem?

Re: [NEW] REST search, JSON results

Posted: Wed Apr 04, 2018 6:32 am
by kioo123
Hi Oss,

Thank you. This actually worked.
Another method is this http://www.misterion.ru/2012/10/qt-qml-user-agent.html
But i prefer yours instead. :D

Re: [NEW] REST search, JSON results

Posted: Wed Apr 18, 2018 10:47 pm
by samsamsam
Hi Oss,

I try this new API:
https://rest.opensubtitles.org/search/i ... /season-1/

But unfortunately, when I try to download subtitles I have problem with reCaptcha v2.
When I try to download for example:
https://dl.opensubtitles.org/en/downloa ... 3226300.gz

the server return:
HTTPError: HTTP Error 429: Unknown HTTP Status

add redirect me to:
http://www.opensubtitles.org/en/captcha ... 3226300.gz
and show reCaptcha v2 checkbox.

I understand why you protect opensubtitles.org with reCaptcha v2 and I am not requesting to disable it , but with currect setting of the reCaptcha v2 my application is not able to handle it.
I asked you in this thread
https://forum.opensubtitles.org/viewtop ... &start=180
to provide possibility to solve reCaptcha v2 without javascript. So, my application will be able to show reCaptcha v2 task to the end user for solving it.

Could you please check this?

Thank you,
SamSamSam

Re: [NEW] REST search, JSON results

Posted: Sat Apr 21, 2018 5:14 pm
by samsamsam
Any answer please?

There is some hole in the re-captcha v2 protection on this site, so I can use it to workaroud these problems.
But I don't want to do this, because I have respect for opensubtitles.org idea. I hope that you can take a look on the problem with reCaptcha v2 and change it settings.

Re: [NEW] REST search, JSON results

Posted: Mon Apr 23, 2018 8:01 am
by oss
429 http code is meant "too many requests", so you have to calm down.

Your IP can be in captcha mode, so in private message you can send me what are you doing exactly, send me urls where are you getting captcha and also your IP, so I can check

The problem with REST is that you are unauthenticated.

Re: [NEW] REST search, JSON results

Posted: Mon Apr 23, 2018 9:01 am
by samsamsam
For sure I did not made to many requests.

Total numbers of downloaded subtitles files in this month are less than 30.
Your IP can be in captcha mode, so in private message you can send me what are you doing exactly, send me urls where are you getting captcha and also your IP, so I can check
I am not doing anything special. I already wrote when I got reCaptcha v2: when I try download subtitles from link returned via REST API.

The problem is not me. I can simple work around this problem. You will check my IP and remove it from reCaptcha database, but this is not what I am asking. I am a developer. I know that many users have such problems I get report form them.
So, I am not asking about solution for me, but general solution to give possibility to solve reCaptcha without javascript.
.
It can be done from reCaptcha configuration.

Please read reCaptcha documentation:
https://developers.google.com/recaptcha ... pt-enabled

Here you have a quote:
Does reCAPTCHA support users that don't have JavaScript enabled?

reCAPTCHA can only provide the optimal experience in terms of security and usability with JavaScript enabled. However, if supporting users who have disabled JavaScript is important for your site, you can enable the alternative challenge with the following steps. Navigate to the admin console and move the security preference slider to "easiest for users". Keep in mind that with this setting reCAPTCHA won't be able to use all of its security features.
My plugin is able to handle reCaptcha task in fallback mode and present reCaptcha task to end user to select valid pictures, but the reCaptcha widget must be configured to allow fallback mode.