Improve Schedules Direct support

[danielk]

This ticket tracks adding Schedules Direct support to MythTV for the trunk and 0.20-fixes.

Steps,

• Basic support for new URLs [done]
• Port Frontend Channel Editor to new lineup management API
• Allow multiple mythtv lineups to share a single DD lineup [done]
• Remove Labs support after 9/1
• Revise handling of the new "new" XMLTV tag [done]

[danielk]

Patch against head for basic Schedules Direct support (portions by Greg Estabrooks), lightly tested.

[danielk]

Same patch backported to 0.20-fixes

[danielk]

Adds basic caching of DD downloaded data.

[danielk]

The updated version of the patch for the trunk adds basic caching of the DD downloaded data. The current patch creates the cache files based on the date, I plan to change this to create unique files per mythfilldatabase run. This caching makes multiple DD based lineup updates per run more efficient.

You can create two MythTV "Video Sources" with the same DataDirect based grabber, the same login info, and the same lineup, but set up two different MythTV channel lineups (left as an exercise for the reader for now). As long as the DataDirect lineup contains all the channels contained in the two or more MythTV lineups, "mythfilldatabase --remove-new-channels" will download the data once and then fill in the listings for all your MythTV lineups while not messing up any of your mythtv channel lineups.

So what's the downside? Well if any new channels are added upstream they will not be automatically added to your lineup, also the "Fetch channels from listings source" button in mythtv-setup will fetch all the channels on a shared analog+digital DataDirect source, including the digital ones you probably can't tune. You will have to delete channels in mythtv-setup's Channel Editor which don't belong; or, configure the DataDirect lineup editor with just the analog channels before setting up an analog card, and then adding back the digital channels before doing anything with the digital cards.

Although most of my testing has been with Schedules Direct, when speaking of DataDirect in this comment I am referring to both Schedules Direct and TMS Labs.

[danielk]

Updated patch, this version should maintain the DD cache across local day boundaries.

[xris]

We'll need a one-time db update of:

UPDATE recorded
   SET programid=CONCAT(SUBSTRING(programid, 1, 2),
                        '00',
                        SUBSTRING(programid, 3))
 WHERE length(programid) = 12;

UPDATE oldrecorded
   SET programid=CONCAT(SUBSTRING(programid, 1, 2),
                        '00',
                        SUBSTRING(programid, 3))
 WHERE length(programid) = 12;

Though we should probably add a similar edit to all imported Labs data for the transition period, too.

[danielk]

Untested patch which adds program id updates to 0.20-fixes patch

[danielk]

Untested update for head patch which adds program id updates

[danielk]

Updated patch, fixes extra FixProgramID runs on startup for 0.20-fixes, also updates program id in program table.

[danielk]

Updated patch, fixes extra FixProgramID runs on startup for trunk, also updates program id in program table.

[danielk]

Updated 0.20-fixes patch to the latest filldata.cpp changes in svn.

[danielk]

(In [14231]) Refs #3842. Adds basic support for Schedules Direct to fixes.

This has fixed all the issues reported in the last few days; but problems may remain, please post any problem reports to the #3842 ticket.

The frontend channel editor does not work with Schedules Direct yet.

You will need to recompile your plugins, as the binary revision has changed.

[danielk]

(In [14232]) Refs #3842. Adds basic support for Schedules Direct to trunk.

This has fixed all the issues reported in the last few days; but problems may remain, please post any problem reports to the #3842 ticket.

The frontend channel editor does not work with Schedules Direct yet.

You will need to recompile your plugins, as the binary revision has changed.

Multiple MythTV lineups per Schedules Direct or Labs lineup is now supported. However, in order to do this you must always run mythfilldatabase with the "--remove-new-channels" option after you set up your channels. A future commit will add some more smarts so that you do not need to do this. Please see the #3842 ticket for more details.

[Axel.Thimm@…]

Should I rebuild fixes & trunk packages at this time? :)

[gtgj@…]

I am getting a segfault when it's trying to cache the listings data. This is with r14238 compiled from source on Fedora 7 x86_64. I am attaching the log and gdb stack trace.

[gtgj@…]

mythfilldatabase r14238 segfault

[anonymous]

Replying to Axel.Thimm@ATrpms.net:

Should I rebuild fixes & trunk packages at this time? :)

Axel, I'd wait if I were you, we'll be releasing 0.20.2 probably within the week. Best to give 2 or 3 days for the SD stuff to be properly tested and any bugs worked out.

[anonymous]

Replying to anonymous:

Replying to Axel.Thimm@ATrpms.net:

Should I rebuild fixes & trunk packages at this time? :)

Axel, I'd wait if I were you, we'll be releasing 0.20.2 probably within the week. Best to give 2 or 3 days for the SD stuff to be properly tested and any bugs worked out.

(BTW who is anonymous in comment:8?)

I'll be traveling across Europe in a week's time, so sooner is better for me. If there are no expected regressions in the current trunk and fixes code (that is to say not more expected than at any other time I'm cutting a package of the fixes/trunk code) then I'd rather see an early green light with the possibility of having to ship an updated package later if something isn't as smooth as expected.

Furthermore the packages could help finding and fixing bugs.

[danielk]

Debugging patch for gtgj

[danielk]

gtgj, can you attempt to reproduce the segfault with the attached debug patch? The log output should give us a hint as to what went wrong.

Axel, I need to fix the reported segfault, and introduce caching to 0.20-fixes. Hopefully this will happen today, then we need a few days to make sure there are no show stopper bugs. We just started beta testing yesterday...

[Axel.Thimm@…]

Replying to danielk:

Axel, I need to fix the reported segfault, and introduce caching to 0.20-fixes. Hopefully this will happen today, then we need a few days to make sure there are no show stopper bugs. We just started beta testing yesterday...

OK, I only started packaging up trunk, I suggest to let the trunk packages out to testers asap and see what else breaks. I'll hold off from the stable packages until you give final green light :)

[danielk]

(In [14243]) Refs #3842. Backports DD caching to 0.20-fixes.

Packagers, please wait a few days before packaging; we've got one bug report involving a segfault that we are still trying to diagnose.

Note: This changes the binary revision and so requires plugins to be rebuilt.

[anonymous]

Replying to danielk:

[...]

Here's the log with the debug patch.

[gtgj@…]

mythfilldatabase segfault trace with debug patch

[Isaac Richards]

Looks like 'in' is keeping the old (closed by that pclose) 'fp' pointer around.

[Isaac Richards]

Should fix crash when caching data..

[Isaac Richards]

gtgj, please try the newly attached patch - it should fix your crash.

[danielk]

New dbg patch

[danielk]

gtgj, Isaac's patch should work, but I found another similar problem as well which is addressed in 3842-head-dbg-2.patch. QFile does not close a file or pipe when you call QFile::close() after opening with a FILE pointer, so this function has been leaking file descriptors for a while. We were also closing the FILE pointer as a pipe regardless of whether it was a pipe or a file (pipes must be closed with pclose and files with fclose, otherwise they are the same).

[danielk]

(In [14244]) Refs #3842. Some small fixes to util.{cpp,h}

• properly capitalize "copy" in doxygen docs
• add error reporting to copy function
• use a proper template as the default for createTempFile

[danielk]

(In [14245]) Refs #3842. Add error reporting to util's copy function.

[gtgj@…]

(Sorry for the delay.) I tried the dbg-2 patch and it seems to work fine. Thanks.

[danielk]

I committed the fix for the segfault in [14247] and [14248].

[danielk]

Adds code to handle the new "new" XML tag in DD

[danielk]

Adds code to handle the new "new" XML tag in DD for 0.20-fixes

[danielk]

(In [14261]) Refs #3842. Adds support for new new XML tag to trunk.

[danielk]

(In [14266]) Refs #3842. Adds support for new "new" XML tag to 0.20-fixes.

[danielk]

Prevents digital channels from being added to analog lineups by default.

[danielk]

The new patch "3842-multilineup-head-v1.patch" allows you to have an analog card and a digital card sharing the same OTA or cable feed without needing to run mythfilldatabase with the "--remove-new-channels" parameter. With this patch MythTV filters out the digital channels in your DD lineup when inserting channels from your DD lineup into an analog source unless you use the new "--do-not-filter-new-channels" parameter. (This isn't needed for digital sources, because there channels are only inserted by a channel scan.)

If you are using Firewire with an STB or capturing from an STB using s-video/composite out, then MythTV can't tell which new channels in the DD lineup belong to what card and you will still need to run mythfilldatabase with "--remove-new-channels" if sharing the same DD lineup in two MythTV lineups (aka "Video Sources").

This patch is more verbose than the final changeset will be.

[danielk]

Prevents digital channels from being added to analog lineups by default for 0.20-fixes (untested on fixes).

[Axel.Thimm@…]

I'm still holding off from touching the stable rpms. But ATrpms' bleeding and testing repos have fresh trunk and fixes builds available as well as the whole debuginfo stuff to nicely backtrace any potential segfault or other crash.

It would be nice if some of the SD beta testers that use RHEL/CentOS/Fedora could give them a try, preferably both branches, 0.20.x ("testing") and 0.21 trunk ("bleeding").

Please note that these packages are offered for aiding developers and bug hunting. If all you want is a production ready mythtv package set, please stick to the stable repo - the packages will move there once Daniel gives the green light and the packages have passed the QA in "testing".

[danielk]

(In [14274]) Refs #3842. Fixes "mythfilldatabase --only-update-channels" issue in 0.20-fixes.

[danielk]

(In [14276]) Refs #3842. Applies "3842-multilineup-fixes-v1.patch" (sorry about the botched patch upload earlier),

This should be the last change to 0.20-fixes before the 0.20.2 release unless some major bug turns up.

I'll attach the testing log to the ticket. I haven't tested every combination of flags, so if you experience any problems or think some additional combination of flags not noted in testing.txt should be tested please note it on the #3842 ticket.

[danielk]

Testing log

[danielk]

(In [14277]) Refs #3842. Fixes "mythfilldatabase --only-update-channels" issue in head.

[danielk]

(In [14278]) Refs #3842. Applies "3842-multilineup-head-v1.patch" with some of the extra verbose macro's from the patch commented out.

This allows you to have an analog card and a digital card sharing the same OTA or cable feed without needing to run mythfilldatabase with the "--remove-new-channels" parameter. With this patch MythTV filters out the digital channels in your DD lineup when inserting channels from your DD lineup into an analog source unless you use the new "--do-not-filter-new-channels" parameter. (This isn't needed for digital sources, because there channels are only inserted by a channel scan.)

If you are using Firewire with an STB or capturing from an STB using s-video/composite out, then MythTV can't tell which new channels in the DD lineup belong to what card and you will still need to run mythfilldatabase with "--remove-new-channels" if sharing the same DD lineup in two MythTV lineups (aka "Video Sources").

This hasn't been as extensively tested in head as in 0.20-fixes but was originally written against head and has recieved plenty of ad-hoc testing in head.

--This line, and those below, will be ignored--

M mythtv/libs/libmythtv/datadirect.h M mythtv/libs/libmythtv/sourceutil.h M mythtv/libs/libmythtv/datadirect.cpp M mythtv/libs/libmythtv/sourceutil.cpp M mythtv/programs/mythfilldatabase/filldata.cpp M mythtv/programs/mythfilldatabase/main.cpp M mythtv/programs/mythfilldatabase/channeldata.h

[Doug Goldstein <cardoe@…>]

An enhancement suggestion (since you wanted all comments to go to this ticket), pop up a message of some sort for "invalid username/password combination" when the user types a bad username or password combination.

[ChrisH <gchris@…>]

In testing, we've seen several instances where the SchedulesDirect.org option is not available until after all existing video sources are cleared. Other users report immediate access to the option without disturbing the video sources. If there is an "official recommendation", it would serve to avoid a lot of confusion if that recommendation took the form of a note on the first Video Sources page.

[anonymous]

Chris H, It shouldn't matter what order you do things in. If you go into a "Video source setup" with "North America (TMS Labs) (Internal)" selected as the grabber, just tab to that field and hit the left arrow key and that will select "North America (SchedulesDirect.org) (Internal)". When you create a new Video Source this is the first thing in that combo box so you just need to fill in the user id and the password fields, hit [Retrieve Lineups] and then select the appropriate SD lineup.

The MythTV will not automatically convert existing Labs setups to Schedules Direct setups, doing so would break the setups of everyone still using Labs. We will continue to support Labs while it is still available.

[ChrisH <gchris@…>]

I'll have to double check the left arrow trick but mouse clicking the TMS entry gave a drop down list that DID NOT include the Schedules Direct option. That option didn't appear at all until after Video Sources was cleared.

[ChrisH <gchris@…>]

The other underlying problem with this is that a user who has never seen the Schedules Direct option doesn't know that the TMS entry isn't Schedules Direct until he tries to use the grabber and it fails because he's got the wrong id and pw for the URL that gets issued.

[ChrisH]

Okay, my error, I hadn't noticed that the scroll bar on the drop down list was slightly down from the top. Moving it up displayed the hidden Schedules Direct entry, but this still feels like Russian roulette. Exposing a user who has just installed this change and has the expectation of connecting with Schedules Direct to the wrong option initially, and expecting him to know that TMS is not what he is looking for is asking a little much IMHO. It sure fooled me, and I've been at this a while.

[anonymous]

This updates programid in all the tables to be the new 14 character format, but it doesn't update seriesid to the new 10 character format. Shouldn't it?

Something like:

UPDATE recorded SET seriesid=CONCAT(SUBSTRING(seriesid, 1, 2), '00', 
  SUBSTRING(seriesid, 3)) WHERE length(seriesid) = 8;
UPDATE oldrecorded SET seriesid=CONCAT(SUBSTRING(seriesid, 1, 2), '00', 
  SUBSTRING(seriesid, 3)) WHERE length(seriesid) = 8;
UPDATE program SET seriesid=CONCAT(SUBSTRING(seriesid, 1, 2), '00', 
  SUBSTRING(seriesid, 3)) WHERE length(seriesid) = 8;

[danielk]

anon, MythTV does not use the seriesid for scheduling unless you write some custom SQL, so I decided to just let it be for now.

[Robert Kulagowski]

Need to widen the username field in mythfilldatabase Video Source Setup page since it's now going to use email addresses instead of just usernames. Push password field to a new line.

[danielk]

Curtis Stanford reported to me that firewire channels are not loaded automagically. This is just a reminder for me to look into this issue.

[danielk]

Reworks video source validation that is run before inserting channels. Notably firewire connected lineups have their channels inserted with SD with this patch (workaround is to disconnect Video Source for firewire and run mythfilldatabase).

[danielk]

(In [14334]) Refs #3842. Rearranges Data Direct setup panel to widen the username field in 0.20-fixes. Same change was added to trunk in [14333].

[danielk]

Updated patch, this fixes a validation bug that caused channels to be added to unconnected sources in safe mode. And also the first time fetch failing problem reported by Robert K.

[danielk]

(In [14355]) Refs #3842. Fixes validation so that firewire channels can be added without the workaround; this also saves an input connection when you fetch channels (validation needs this to work as intended without invasive changes).

I'm not going to apply this to fixes until it's baked in SVN for a while. There are too many card combinations and connection states for me to be sure I've tested everything and I don't want to have any risk that I'll contaminate fixes with new bugs when so many people are updating.

[danielk]

(In [14397]) Refs #3842. Backports source validation changes from head.

Fixes validation so that firewire channels can be added without the workaround; this also saves an input connection when you fetch channels (validation needs this to work as intended without invasive changes).

This also fixes a bug in 0.20-fixes SourceUtil::IsUnscanable(uint sourceid) which had already been fixed in head; this fix was needed for the firewire channel fetch to work (since it has an "unscanable" input).

[danielk]

(In [14425]) Fixes #3920. Refs #3842. Clears userid and password when switching from a grabber with these to one without it.

This prevents us from trying to grab the lineups from Schedules Direct when the user has switched a former SD video source to EIT or XMLTV.

[danielk]

(In [14426]) Refs #3842. Removes TMS Labs grabber option in svn trunk.

Much of the code is still here, I'll be repurposing the labs specific lineup management code for Schedules Direct, then removing the unused portion.

[danielk]

(In [14427]) Refs #3842. Removes TMS Labs grabber option in 0.20-fixes.

[danielk]

(In [14429]) Refs #3842. Refines the handling of the 'new' tag. We only see this tag on SHows and EPisodes, never on SPorts (even if 'classic baseball games') or MoVies, so with this change we only set previouslyshown for SHows or EPisodes.

[danielk]

(In [14430]) Refs #3841. Backports [14429] to 0.20-fixes, this refines the handling of the 'new' tag.

[danielk]

I just added two patches to the ticket.

The first, 3842-remove-new-handling-fixes-v1.patch, removes all handling of the new "new" tag in 0.20-fixes. It's causing more problems than it solves.

The second, 3842-add-new-column-trunk-v1.patch, adds an isnew column to the program table which allows us to handle the new "new" tag directly.

After these are applied, the marking of repeats is still incorrect in it's handling of previouslyshown for SHows and SPorts. But this is an old bug, and I've e-mailed Bruce and David for some guidance on how to proceed to fix it.

These patches have only been lightly tested, and the patch against trunk updates the DB schema.

[danielk]

(In [14486]) Refs #3842. Removes 'new' XML tag conversion to 'isrepeat' for DataDirect in fixes.

[danielk]

(In [14487]) Refs #3842. Removes 'new' XML tag conversion to 'isrepeat' for DataDirect in trunk.

[danielk]

Updates add "isnew" column patch to current trunk revision

[anonymous]

Replying to danielk:

Hi!

After these are applied, the marking of repeats is still incorrect in it's handling of previouslyshown for SHows and SPorts. But this is an old bug, and I've e-mailed Bruce and David for some guidance on how to proceed to fix it.

Wouldn't taking into account your isnew column in the program table fix at least partially the marking of repeats (ie not mark as repats programs which are considered new)?

Nick

(see "Why do "news" programs appear as reruns?" in SD forums, I thought the problem was caused by the DD data...)

[danielk]

Few remaining issues can be fixed after 0.21

[danielk]

This cleans up the housekeeper code which starts mythfilldatabase -- more bullet proof in case of db problems..

[danielk]

(In [17231]) Refs #3842. Removes artificial cap of REFRESH_MAX for the number of days of listing data that can be pulled by mythfilldatabase.

[danielk]

any additional improvements should get their won tickets..