Forum rules
Under no circumstances is spamming or advertising of any kind allowed. Do not post any abusive, obscene, vulgar, slanderous, hateful, threatening, sexually-orientated or any other material that may violate others security. Profanity or any kind of insolent behavior to other members (regardless of rank) will not be tolerated. Remember, what you don’t find offensive can be offensive to other members. Please treat each other with the kind of reverence you’d expect from other members.
Failure to comply with any of the above will result in users being banned without notice. If any further details are needed, contact: “The team” using the link at the bottom of the forum page. Thank you.
User avatar
oss
Site Admin
Posts: 5879
Joined: Sat Feb 25, 2006 11:26 pm
Contact: Website

[ADMIN] Changes in API response, APPs error

Thu Oct 27, 2016 8:22 pm

Hi,

we made some minor updates in response for API, and we see some APPs are broken for now, for unknown reason. Changes made:
  • removed pretty print responses. Now responses are without whitespaces (still valid xml, save 50% of data in some cases). Example https://api.opensubtitles.org/xml-rpc (check the source - it is "all together")
  • compression deflate is preferred now (slightly better compression ratio than gzip)
  • API is served through CloudFlare. Some clients doesn't support accept-encoding gzip/deflate and CF is doing great job here.
  • some X- http headers was added for internal logging
All responses are valid, but some clients can not handle them for unknown reason for now. Please make an update to your clients and write here what was the problem.

So far problematic clients:
VLSub - https://github.com/exebetche/vlsub/issues/143 - temporary workaround possible, problem is Transfer-Encoding: chunked
SolEol - unknown reason (can accept just pretty print xml-rpc?)

some technical stuff:
HTTP request WITHOUT ACCEPT_ENCODING:
Current (new) http header response looks like this (using cloudflare)

Code: Select all

< HTTP/1.1 200 OK < Date: Thu, 27 Oct 2016 18:51:31 GMT < Content-Type: text/xml;charset=UTF-8 < Transfer-Encoding: chunked < Connection: keep-alive < Set-Cookie: __cfduid=d7066c70fe9dde24cac3c4e20e01a1b9f1477594290; expires=Fri, 27-Oct-17 18:51:30 GMT; path=/; domain=.opensubtitles.org; HttpOnly < Set-Cookie: PHPSESSID=e6g89l8lqdbbdm0lu7j20aut7; path=/; domain=.opensubtitles.org; HttpOnly < X-Content-Encoding: gzip < X-Uncompressed-Content-Length: 386 < X-Compressed-Content-Length: 205 < Access-Control-Allow-Origin: * < Access-Control-Allow-Methods: GET, POST, OPTIONS < Access-Control-Allow-Headers: Origin,X-Requested-With,Content-Type,Accept,DNT,Keep-Alive,User-Agent,If-Modified-Since,Cache-Control < X-Cache-Backend: web2 < Vary: Accept-Encoding < Age: 0 < X-Cache: MISS < Server: cloudflare-nginx < CF-RAY: 2f8856fd43fd03b8-SIN
Old header looks like this:

Code: Select all

< Set-Cookie: PHPSESSID=e6g89l8lqdbbdm0lu7j20aut7; path=/; domain=.opensubtitles.org; HttpOnly < Access-Control-Allow-Origin: * < Access-Control-Allow-Methods: GET, POST, OPTIONS < Access-Control-Allow-Headers: Origin,X-Requested-With,Content-Type,Accept,DNT,Keep-Alive,User-Agent,If-Modified-Since,Cache-Control < Content-type: text/xml;charset=UTF-8 < Content-Length: 386 < Date: Thu, 27 Oct 2016 18:54:14 GMT < X-Cache-Backend: web3 < Age: 0 < X-Cache: MISS < Accept-Ranges: bytes < Connection: keep-alive
HTTP request WITH ACCEPT_ENCODING: Accept-Encoding: deflate, gzip
New header looks like this:

Code: Select all

< Date: Thu, 27 Oct 2016 19:00:14 GMT < Content-Type: text/xml;charset=UTF-8 < Content-Length: 205 < Connection: keep-alive < Set-Cookie: __cfduid=d3b1fcb018704e3f00dc9f2af131d6e0e1477594810; expires=Fri, 27-Oct-17 19:00:10 GMT; path=/; domain=.opensubtitles.org; HttpOnly < Set-Cookie: PHPSESSID=e6g89l8lqdbbdm0lu7j20aut7; path=/; domain=.opensubtitles.org; HttpOnly < Content-Encoding: gzip < X-Content-Encoding: gzip < X-Uncompressed-Content-Length: 386 < X-Compressed-Content-Length: 205 < Access-Control-Allow-Origin: * < Access-Control-Allow-Methods: GET, POST, OPTIONS < Access-Control-Allow-Headers: Origin,X-Requested-With,Content-Type,Accept,DNT,Keep-Alive,User-Agent,If-Modified-Since,Cache-Control < X-Cache-Backend: web3 < Vary: Accept-Encoding < Age: 0 < X-Cache: MISS < Accept-Ranges: bytes < Server: cloudflare-nginx < CF-RAY: 2f8863af96b91123-SIN
Old header looks like this:

Code: Select all

< Set-Cookie: PHPSESSID=e6g89l8lqdbbdm0lu7j20aut7; path=/; domain=.opensubtitles.org; HttpOnly < Content-Encoding: gzip < X-Content-Encoding: gzip < X-Uncompressed-Content-Length: 386 < X-Compressed-Content-Length: 205 < Access-Control-Allow-Origin: * < Access-Control-Allow-Methods: GET, POST, OPTIONS < Access-Control-Allow-Headers: Origin,X-Requested-With,Content-Type,Accept,DNT,Keep-Alive,User-Agent,If-Modified-Since,Cache-Control < Content-type: text/xml;charset=UTF-8 < Content-Length: 205 < Date: Thu, 27 Oct 2016 18:57:15 GMT < X-Cache-Backend: web2 < Vary: Accept-Encoding < Age: 0 < X-Cache: MISS < Accept-Ranges: bytes < Connection: keep-alive

User avatar
eduo
Posts: 716
Joined: Sat Feb 10, 2007 1:40 am
Location: Information Technology
Contact: ICQ Website Yahoo Messenger

Re: [ADMIN] Changes in API response, APPs error

Thu Oct 27, 2016 11:34 pm

SolEol is one of the clients that is failing. XML structure has changed and since the current version had been built looking for the specific structure, it now has changed.

I'll see about issuing an update, but I fear I'll have to change the whole XML interpretation infrastructure. Is it safe to move to JSON or is that option not recommended? JSON is also smaller which saves bandwidth and time.
http://eduo.info/
[url=http://eduo.info/soleol/]OpenSubtitles from your desktop: SolEol for Mac/Windows/Linux[/url]
[url=http://forums.plexapp.com/index.php?showtopic=325&st=0&p=2480&#entry2480]My current episode processing work flow[/url].

User avatar
oss
Site Admin
Posts: 5879
Joined: Sat Feb 25, 2006 11:26 pm
Contact: Website

Re: [ADMIN] Changes in API response, APPs error

Fri Oct 28, 2016 4:45 am

Hi Eduo,

sorry to hear that SolEol is broken now. Thats why I made this topic. I would like to know, why it is failing. When pretty printed XML will be back, it will work again ? Did you use some XML parsing library (XMLRPC) ? What is exactly problem.

In theory, when pretty print is problem, I could make list of User Agents, where pretty print will be enabled for backward compatibility and there should be update released to accept 'nowhitespace' responses.

For json responses - thats a project for next version of opensubtitles.

User avatar
eduo
Posts: 716
Joined: Sat Feb 10, 2007 1:40 am
Location: Information Technology
Contact: ICQ Website Yahoo Messenger

Re: [ADMIN] Changes in API response, APPs error

Fri Oct 28, 2016 10:11 pm

Actually, I'm fixing it now. I'll release a new version of the old client (which is the one most used anyway). I'm just testing all scenarios.

The problem is that in that client I was calling the tree of the XML directly. This means calling childs and parents. This structure has changed now (not by much, but where before we had a child(1) now the actual value is in child(1) in most responses).

It's a silly thing, but obviously it breaks everything up. 10 years ago I probably should've used a better parser but I honestly never thought SolEol would be that much popular :)

I'm getting dozens of reports each day, but fixing the XML engine is too much, so what I'm doing is fixing the mappings to the responses now are correct. While I'm at it I'll probably fix a couple more things that were failing in Windows and Linux.

Eduo
http://eduo.info/
[url=http://eduo.info/soleol/]OpenSubtitles from your desktop: SolEol for Mac/Windows/Linux[/url]
[url=http://forums.plexapp.com/index.php?showtopic=325&st=0&p=2480&#entry2480]My current episode processing work flow[/url].

User avatar
oss
Site Admin
Posts: 5879
Joined: Sat Feb 25, 2006 11:26 pm
Contact: Website

Re: [ADMIN] Changes in API response, APPs error

Sat Oct 29, 2016 5:21 am

Hi Eduo,

please release new version (number in useragent), so I can make some hard fix in my code.

The structure of xml-rpc was not changed, just spaces (pretty print) was removed. Thats why I am wondering whats going on.

I can in theory make in code, if SolEol <= x.xx then pretty print xml, so it will be compatible

User avatar
eduo
Posts: 716
Joined: Sat Feb 10, 2007 1:40 am
Location: Information Technology
Contact: ICQ Website Yahoo Messenger

Re: [ADMIN] Changes in API response, APPs error

Sat Oct 29, 2016 11:25 am

I've released version 1.5.1 that can deal with the new scheme. I've updated the site and notified over twitter.

It's too late now to change the agent :)
http://eduo.info/
[url=http://eduo.info/soleol/]OpenSubtitles from your desktop: SolEol for Mac/Windows/Linux[/url]
[url=http://forums.plexapp.com/index.php?showtopic=325&st=0&p=2480&#entry2480]My current episode processing work flow[/url].

User avatar
oss
Site Admin
Posts: 5879
Joined: Sat Feb 25, 2006 11:26 pm
Contact: Website

Re: [ADMIN] Changes in API response, APPs error

Sat Oct 29, 2016 12:48 pm

OK, so I will just let it be. if you wrote me, which versions was affected (e.g. less then 1.5.1) then I could fix it probably, so they will work.

VinceGreen
Posts: 4
Joined: Sat Oct 29, 2016 4:20 pm

Re: [ADMIN] Changes in API response, APPs error

Sat Oct 29, 2016 4:34 pm

Hi, for the past few days I haven't been able to download any subtitles using the OpenSubtitles API.
I use Server from the xmlrpclib package in python. Everything seems to work like it should (login, logout), with the exception that whatever query I use to SearchSubtitles, I receive a response that has an empty datafield:

Code: Select all

server.SearchSubtitles(token, [{'sublanguageid': language, 'query': searchName}]) ==> {'status': '200 OK', 'seconds': 0.009, 'data': []}
I have reviewed my code and have no idea what I could be doing wrong. Is this a problem on your end? Because I have also tried the subtitle download feature from SMPlayer which used to work, but now also returns zero results. The same goes for the VLSub extension for VLC media player.

User avatar
oss
Site Admin
Posts: 5879
Joined: Sat Feb 25, 2006 11:26 pm
Contact: Website

Re: [ADMIN] Changes in API response, APPs error

Sat Oct 29, 2016 5:22 pm

thanks for info. Can you specify the parameters which you are passing ?

User avatar
eduo
Posts: 716
Joined: Sat Feb 10, 2007 1:40 am
Location: Information Technology
Contact: ICQ Website Yahoo Messenger

Re: [ADMIN] Changes in API response, APPs error

Sat Oct 29, 2016 6:32 pm

thanks for info. Can you specify the parameters which you are passing ?
Also if at all possible a sample XML response file.

OSS: Do you have a sample XML file from the old API? I remember we uploaded a few ones to Trac but I can't find them now. I wanted to compare both XMLs so you could see where things changed that would break some very strict XML parsers.
http://eduo.info/
[url=http://eduo.info/soleol/]OpenSubtitles from your desktop: SolEol for Mac/Windows/Linux[/url]
[url=http://forums.plexapp.com/index.php?showtopic=325&st=0&p=2480&#entry2480]My current episode processing work flow[/url].


VinceGreen
Posts: 4
Joined: Sat Oct 29, 2016 4:20 pm

Re: [ADMIN] Changes in API response, APPs error

Sun Oct 30, 2016 2:23 pm

Sorry for my late response.

I'm not passing any parameters. Using the library I simply execute the following commands which used to work (and now seemingly still do, except for the data part).

Code: Select all

from xmlrpclib import Server server = Server('http://api.opensubtitles.org/xml-rpc') server.LogIn('', '', 'en', '0penSubDL') server.SearchSubtitles(token, [{'sublanguageid': language, 'query': searchName}]) server.LogOut(token)
And they return (minus the keyword at the front):

Code: Select all

server: <ServerProxy for api.opensubtitles.org/xml-rpc> login: {'status': '200 OK', 'seconds': 0.008, 'token': '4s92folu49nc8p3d4730is5i40'} search: {'status': '200 OK', 'seconds': 0.003, 'data': []} logout: {'status': '200 OK', 'seconds': 0.001}
I think the xmlrpclib automatically handles the xml and presumably converts it into json. I simply get returned the json. The json snippet I returned is the complete response I get when executing SearchSubtitles. I have zero interaction with the xml myself.
So perhaps the library is doing something that doesn't work with the new version of the api. I don't know. I'm not quite sure what actually happens there, so I can't comment.

You can find the source code of (what I believe to be) the library here if you are inclined to look into it: http://effbot.org/media/downloads/xmlrpclib-1.0.1.zip
I honestly am not. :wink:

I'll also say this, I work on this in my free time, so it's going to take a while before I refactor my code as I'm busy nowadays. It's not really a problem though. Seeing the header examples in the OP, I do have the feeling if I stop relying on the library I should be able to work it out on my own.

Cheers

User avatar
oss
Site Admin
Posts: 5879
Joined: Sat Feb 25, 2006 11:26 pm
Contact: Website

Re: [ADMIN] Changes in API response, APPs error

Sun Oct 30, 2016 7:04 pm

can you install some kind of http sniffer (which OS you are using?). I really need to see REQUESTS and RESPONSE, so I can debug more.

If you not give me parameters, I can not debug it.

VinceGreen
Posts: 4
Joined: Sat Oct 29, 2016 4:20 pm

Re: [ADMIN] Changes in API response, APPs error

Sun Oct 30, 2016 10:54 pm

I've capture the requests and responses for the LogIn, SearchSubtitle and LogOut respectively. I run linux and used wireshark.
I put it in a pastebin as it is quite a lot of text: http://pastebin.com/M6nkk48Z

User avatar
oss
Site Admin
Posts: 5879
Joined: Sat Feb 25, 2006 11:26 pm
Contact: Website

Re: [ADMIN] Changes in API response, APPs error

Mon Oct 31, 2016 4:50 am

great, thanks for log. There was small typo error in SearchSubtitles - queries was not properly parsed. Now I am getting back results, it should work also for you.

Return to “Developing”

Who is online

Users browsing this forum: Ahrefs [Bot] and 32 guests