EIT rewrite

[Stuart Auchterlonie]

This ISN'T a wishlist. This is what we need to do to rewrite the EIT handling for 0.20

#778 - Astra 19.2E EPG aka. remove Kristian Kalweit fix

#892 - Handling channels with high EIT data rates.

#913 - Handling of EIT data with additional data

#1025 - Handling of unicode characters within EIT data.

Split all eitFixup routines out of siparser. Use them on EIT data only after we have decided that this is new/updated information.

Keep track of which events we have sent to the database. ie. do our own caching of EIT data, getting rid of that currently done for events in siparser (also #943) (GreyFoxx has done most of this already)

EIT data prioritization. which is better EIT or xmltv? Within EIT, lower table numbers always seem to be better data. 0x4e & 0x4f now & next, 0x50 & 0x51 future, 0x60 & 0x61 far future, and other longer term EIT data from various providers

Fixup routines sometimes try and put a subtitle in the DB that exceeds the length of the subtitle field in the DB, so silent truncation of information occurs.

[mythdev@…]

you probably knew this anyway - but just in case it helps with any design decisions:

a) In UK dvb-t the EITFixupStyle2 function can account for ca 40% of the total processing time in LiveTV - its called ~40 times/sec - those Qt regexes really hurt in this function :)

b) For dvb-t I don't believe the scan needs to look at every channel - just every multiplex

I was about to look see if it was practical to skip ParseDVBEIT for say 10% of the packets or even to skip it completely during recording/livetv - ymwv!

[danielk]

Reassigning, so this shows up on my ticket list

[danielk]

(In [9062]) References #1035. Cleanup of EITFixUp.

This keeps most of the regular expressions around across calls to the fixup routines so that we don't need to depend on Qt's regular expression caching.

[danielk]

(In [9074]) References #1035. SIParser refactoring.

[danielk]

(In [9098]) References #1035. Eliminates the last ATSC EIT descriptor parsers in siparser.

[danielk]

(In [9099]) References #1035. Removes DVB Content Descriptor parser from SIParser.

[danielk]

(In [9100]) References #1035. Removes DVB Component Descriptor parser from SIParser.

[danielk]

(In [9101]) References #1035. Removes DVB Short and Extended Event Descriptor parsers from SIParser.

[danielk]

(In [9103]) References #1035. Couple missing pieces from [9100].

[danielk]

(In [9110]) References #1035. Removes DVB EIT parser from SIParser.

[danielk]

(In [9111]) References #1035. Removes unused ConvertDVBDate() method.

[danielk]

(In [9112]) References #1035. Removes STTHandler. The System Time table table is always in a single segment, and we want to see the updates...

[danielk]

(In [9116]) References #1035. Uses ATSCStreamData to call MGT, VCT, STT, and ETT handlers.

[danielk]

(In [9117]) References #1035. Uses ATSCStreamData to call HandleEIT().

[danielk]

(In [9118]) Fixes #1375, References #1035. Allow each A/V stream to use multiple Conditional Access methods.

Apparently this is used in Australia. I'm not sure if they use different methods for the video, audio, and data streams, or if they just list all the supported methods and hope one is sticks. Multiple CA methods per PMT was already supported.

[danielk]

(In [9135]) References #1035, fixes some NIT descriptor parsing bugs.

[danielk]

(In [9139]) References #1035. Removes a few more descriptor parsers from siparser.

This should also allow a DVB scans to pick up a few more transports correctly.

[danielk]

(In [9227]) References #1035. DVB EIT versioning + fix for PMT pull.

The EIT versioning is the bulk of the patch, but there were also some problems with the PMT handling in [9224] discovered by Oscar which this should fix.

[danielk]

(In [9228]) References #1035 and closes #1427, by applying patch from #1427.

This implements an event signature cache, so that we avoid processing an event again during any one EIT scan.

This should lower the memory requirements and CPU load of the EIT Scan somewhat.

[danielk]

(In [9664]) References #1035. Minor EIT cleanup & ATSC EIT bug fix.

This removes some now unused signals and slots, and removes all signals and slots in EITHelper.

This also processes the MGT and VCT in siparser. This was disabled by commit a week or two ago, and breaks EIT in ATSC Land.

[danielk]

(In [9668]) References #1035. ATSC EIT refactor.

This moves most of the EIT parsing for ATSC out of SIParser and into the eit classes. This also adds language awareness and preference to the ATSC EIT parsing.

[chris [@…]

Hi,

I am getting a crash on startup with 9664-8. I've tried 9663 which is fine. Looking at the GDB output, it appears to be a problem in the EIT scanner.

I've attached some backtraces from 9666 and 9666.

HTH

Chris

[danielk]

(In [9669]) References #1035. This moves most of the EIT parsing for DVB out of SIParser and into the eit classes.

I haven't ported the fixup functions over yet. The new structure depends on being able to de-link eit event processing from the channel/transport we are currently on in the SIParser. But a lot of the "PrivateType" stuff for the fixups is not cached.

[danielk]

(In [9671]) References #1035. Restores EITFixUp's for DVB EIT.

[danielk]

(In [9672]) References #1035. Fixes a copy-n-paste error in program credits update in [9671].

[danielk]

(In [9673]) Closes #1667, References #1035. Fix for off-by-one error in myth listings category lookup found by Janne.

[danielk]

(In [9674]) Fixes #1668. References #1035. Fixes copy-n-paste error.

What should have been the index in the loop was copied from another part of the function that used a result of the loop.

[danielk]

(In [9688]) References #1035. Removes --disable-dvb-eit compile-time option.

This moves the eit stuff outside the DVB umbrella, it is now almost usable by the HDTVRecorder (for those HD-2000 folks).

[danielk]

(In [9689]) References #1035, Fixes #892. Implements rate limiting for EIT events.

[danielk]

(In [9701]) References #1035. Fixes timezone conversion for ATSC EIT events. This was messed up earlier this week (timezone compensation was applied twice).

[danielk]

(In [9743]) References #1035, References #1701.

You may need to do a distclean. This removes signals and slots from the channels and recorders.

I've tested this with every recorder I have access to, but I don't have access to a firewire or dbox2 recorder.

If you experience any problems please do a distclean before reporting the problem.

The reasons for getting rid of signals and slots in non gui elements are because they require delayed deletes and other extra complexity to avoid segfaults, and because they are not thread safe and are incompatible with virtual inheritence and multiple inheritence.

[danielk]

(In [9763]) refs #1035, refs #1688. Preferred language fix for DVB EIT.

[danielk]

(In [9767]) Refs #1035, refs #1688. Always do some basic fixups with DVB EIT.

[danielk]

(In [9798]) Refs #1035. Refs #1049. This is a continuation of the refactor started in [9790].

This makes it possible for DVBRecorder and SIParser to use a DVBStreamData in addition to a ATSCStreamData.

This allows us to move more of the DVB specific EIT stuff from SIParser to DVBStreamData.

I also referenced #1049, because this lets us use a DVBStreamData in DVBSignalMonitor and then pass it along to both the DVBRecorder and DVBSIParser; which simplifies the implementation of NIT/SDT monitoring.

This also adds SetStreamData() and GetStreamData() stubs to DTVRecorder, this allows us to get rid of a few of the ifdef's in TVRec. TVRec has also been modified to use a DVBStreamData rather than an ATSCStreamData when starting a signal monitor for a non-ATSC digital channel.

[danielk]

(In [9807]) Refs #1035. More EIT refactoring.

This removes all the EIT handling from SIParser except for the pid filtering function.

[danielk]

(In [9816]) Refs #1035. More EIT refactoring for EIT branch.

The major change is the removal of the DVBSIParser and the code it depends on. DTVSignalMonitor picks up it's small remaining role in active EIT scanning. This in turn enables EIT scanning for all recorders with a DTVSignalMonitor based signal monitor.

This patch looks a lot bigger than it is. Many of the changes are deletions of files we no longer use now that [DVB]SIParser is gone.

By replacing the dvbdevice() function, which was our only remaining dependence on dvbdev aside from dvbci, we can also handle more than 4 DVB devices now. Admitedly this is not a common problem, but there is no good reason for this limitation.

[danielk]

(In [9830]) Refs #1035. Adds a convenience methods that allow you to query which channels are cached in the stream data.

[danielk]

(In [9831]) Refs #1035. Refactors DVB recording a bit to work better with StreamData classes (which is being used for more generalized EIT handling).

In order allow the recorder to monitor the PAT & PMT for the stream this makes DVBRecorder and not get confused by multiple PMTs on a pid this moves the HandlePAT() and HandlePMT() methods from the DVBChannel to the DVBRecorder.

This moves some the channel table DB stuff from TVRec to ChannelUtil as a cleanup. This also to allows us to more easilly check whether two channels share a multiplex by untie these queries from the current channel in the channel class.

This also adds the other pids in the PAT to the pids being monitored in DVBRecorder, this allows us to more quickly change between channels on the same transport by simply querying if we have the needed PSIP tables and using them if they are present.

So far this has been tested with the HDHomeRun and DVB recorders, but it should also speed up FireWire and HDTV recorder channel changing since the use the StreamData classes.

[danielk]

(In [9841]) Refs #1035, refs #1728. Fixes a cut-n-paste error in DVBRecorder which caused a segfault in DVB Land (w/ patch from Janne).

[danielk]

(In [9844]) Refs #1035. Fixes #1049. Adds SDT monitoring to DTVSignalMonitor in mythtv-eit branch.

What this does is require a matching SDT table for DVB channels before decide that we are on the right transport. This especially helps with DVB-S tuning, but also lets us collect the SDT table in the DTVSignalMonitor which allows us to dependably collect DVB EIT now that the SIParser class is gone.

This also changes the scanning for ATSC/MPEG a bit. We've been inserting a networkid of '1' instead of '0' for these transports for a while. There is a db update in dbcheck to fix existing transports and some changes in siscan to insert these properly in future channel scans.

Please test/debug so that I can merge this with SVN-head soon.

[danielk]

(In [9847]) Refs #1035. merged trunk/mythtv changes r9833:9846 into the mythtv-eit branch.

[danielk]

(In [9957]) Refs #1035. Big EIT branch merge.

This moves the EIT handling out of SIParser and mostly into the DVB and ATSC stream handlers, allowing recorders other than the DVBRecorder to handle EIT without massive code duplication. This also allows us to get rid of a bunch of dependencies on the dvbdev stuff. The only thing we are still using there is the CAM code.

This also fixes several problems in the MPEG PES packet assembly code that were uncovered when we started using it for EIT in addition to using it for the tuning tables.

Since SIParser is gone this also allows us to use the table caching in the stream handler code to speed up intra-transport tuning. If we already have the needed tuning tables we skip the signal monitoring step and begin recording the new channel immediately.

This also fixes various bugs in EIT handling, and lowers the EIT processors memory consumtion further.

This also removes the arbitrary limit of four DVB cards per backend. Now you should be able to add as many DVB devices as your hardware allows.

I recommend a "make distclean".

[danielk]

(In [9973]) Refs #1035. Fixes #1801. Fixes a copy-n-paste error that resulted in an ignored EIT fixup regular expression in the UK EIT fixup.

[danielk]

(In [9976]) Refs #1035. Fixes #1728. Applies Janne's EIT Cache 'save to DB' patch.

This allows the EIT version cache to be save to the DB so that it can be shared amoung recorders on several backends.

[danielk]

(In [10238]) Refs #1035. Refs #1728. Refs #1801. Refs #1884. Refs #1615. EIT branch merge.

• Improves the EIT Fixup functions for the UK and Sweden.
• Improves the EIT version cache (lowering memory and CPU requirements).
• Fixes various small memory leaks and lowers memory requirements with earlier deletes.
• Fixes a spurious error message in filldata with EIT only sources.

[danielk]

(In [10244]) Refs #1035. Undo overly Janne's leak fix for StreamData in tv_rec. It improperly deletes stream data in some cases.

[Stuart Auchterlonie]

(In [10584]) Refs #1035. Sync's mythtv-eit branch back to head.

[Stuart Auchterlonie]

(In [10614]) Refs #1035, increases the wait time in the eit event loop

This increases the wait time in the eit event loop from 200ms to 400ms, which makes the eit processing less aggressive and more like background processing.

[danielk]

(In [11752]) Refs #1035. Several European DVB providers don't set the character set for their EIT listings as required by the DVB standards so we have been applying an override to set it to a sensible value; but, for some reason the 'sensible value' we chose was the Turkish character set. In all likelyhood they are using something closer to the Western European character set, so I've changed the character set appropriately.

The two character sets are very similar, they differ mostly in how they handle some special characters which is why this wasn't caught earlier.

[btsglw@…]

Channel dump as requested by Daniel

[btsglw@…]

DTV_mulitplex dump as requested by Daniel

[anonymous]

Change set 11752 introduced an eit bug with Bell Expressvu. The early eit fixups for the character set is overwriting previous defined fixups.

[Stuart Auchterlonie]

(In [11817]) Refs #1035. Fixes the problem with the Bell Expressvu eit fixups that were introduced in [11752]

[Janne Grunau]

(In [12820]) Refs #2342, #1035. Adds channel looking to eit cache

This should prevent duplicate eit insertions. The channel looks are removed when the master frontend starts or after 24 hours. The eit cache will now written back to the DB after every channel change. I hope this isn't too much stress for anyone's database / backend server. I didn't notice the higher load.

[Janne Grunau]

(In [12831]) Refs #1035. Optimize DVB EIT insertion.

One DVB event information table caries only events of a single channel => look the chanid only once per section up.

Simplifies the eit cache by removing duplicate chanid lookup.

Fixes eit cache issues with the same channel on different sources (reported by Otto Kolsi)

[danielk]

We'll open a new ticket for the next EIT rewrite..