Channel Scanner cleanup

[danielk]

I'm working on making the Channel Scanner more maintainable and fixing some lingering bugs. This ticket is just to track the progress.

DONE

• It is possible to start the scanner before you set the video source in input connections pane. [fixed in patch]
• It is possible to start the scanner from the channel editor, even if no scannable sources exist. [fixed in patch].
• The scan wizard does not prefer the current card and card input if you launch it from the input connections pane [fixed in patch].
• The scan wizard does not prefer the current source if you launch it from the channel editor pane [fixed in patch].
• The scan wizard does not show always you which card and input it will use when scanning [fixed in patch].
• The scan wizard does not handle multiple input cards correctly. [DONE]
• We should make the Analog scanner work more like the normal channel scanners; at the moment it uses a completely different and mostly unmaintained class for scanning which does not play well with XMLTV or DataDirect channel insertion, does not handle inputs correctly, etc. [DONE]

TODO

• The scan wizard classes interact in strange ways: functional classes access private portions of the UI classes, the ScanWizard fires off events just to run functions it could call directly, etc. [partially fixed in patch]
• #2600 Channel scanner stops scanning after finding (C)VCT
  • We need to separate finding channels and inserting them [DONE]
  • We should report channels which are in a channels.conf import, but were not found by the scanner (and perhaps allow these to be inserted anyway).
  • We should report channels which are in the PAT and have a PMT, but do not show up in a VCT and allow the user to insert these. [DONE]
  • We should optionally allow the user to wait for the PAT and PMTs even after a VCT has been found.
• #1866 DVB Scanning - Remaining Issues
  • Tuned Scan does not use the information found in the NIT to find other transports to scan, as it did in previous releases.
  • Other????

[danielk]

Initial patch

[danielk]

(In [11772]) Refs #2695. Fixes a few minor problems with the channel scanner UI and improves the UI/scanner interface.

The problems fixed are:

• It is possible to start the scanner before you set the video source in input connections pane.
• It is possible to start the scanner from the channel editor, even if no scannable sources exist.
• The scan wizard does not prefer the current card and card input if you launch it from the input connections pane.
• The scan wizard does not prefer the current source if you launch it from the channel editor pane.
• The scan wizard does not show always you which card and input it will use when scanning.

The SIScan class itself hasn't been touched in this changeset, I will be fixing some of the functional problems separately.

[aew.mythtv@…]

log of setup import of /etc/channels.conf

[aew.mythtv@…]

Plus answer these questions (in the ticket):

• Are you using DiSEqC?

no

• How many cards is this source connected to?

only one

[danielk]

Commit [11816] should have referenced this ticket: Reworks analog channel scanner to use the standard classes and makes it compatible with datadirect channel updates.

[danielk]

(In [12183]) Fixes #2756. Refs #2695. Allows scanner to work properly when DiSEqC input is connected to a source which is also connected to another DiSEqC input.

Before the refactors in #2695 we did not know which input we were scanning so each input had to be connected to a different source, even if the two sources contained the same channels. Now we pass the inputname to the scanner so that it knows which input it should scan if a source is connected to multiple inputs.

[danielk]

Updated TODO for this tack ticket.

[danielk]

(In [12419]) Refs #2695. Removes some unused parameters in DiSEqC setup.

These were added to try to simplify DiSEqC setup, but it turned out that fine tuning a DiSEqC setup is a common operation for FTA hobbists so I removed the simplifying code. This removes the paramaters that were used to enable and disable the simple input connections panel but are now unused.

[anonymous]

I have some problems concerning the scanner too: In my case mythtv is not able to find the some channels even if they were first imported correctly from a channels.conf file (and after that deleted so that mythtv can do the scanning again to make EPG working on these channels): A scan on all known transponders does not give some "special" channles: The "Premiere Direkt" (and other pay per view) channels are missing.

Maybe there is a workaround to fix the database manualy to get EPG working with these few imported channels which the mythtv scanner does not find. I looked into the mythconverg database and found that the cause for the missing EPG maybe lies in the table "dtv_multiplex". For channels which are impoted from a channels.conf file, the "transportid" and "netwokid" in the corresponding entry of the "dtv_multiplex" seems to be alway NULL in comparison to the entries that were made when mythtv does the scanning. Is it possible to fix the EPG on theses channels by fixing this table? Or which other information is missing in the database to get EPG working? And how can this be fixed by hand?

[danielk]

(In [13256]) Refs #2695. Refs #2909. Refs #3119. Refs #3190. Refs #2830. Refs #2600. This commit creates a branch for working on the channel scanner.

[danielk]

(In [13257]) Refs #2695. Applies channel scanner patch to channel scanner branch.

This is not a working scanner, it will not insert any channels into the DB. What it does is put all the channels it finds into a data structure which can be processed to insert channels into the DB. The idea is to have all the information we can get before we start inserting the channels, this allows us to address problems such as #3119 where all the channels have the same major and minor channel numbers according to the info in the streams and so that info should be ignored.

[danielk]

(In [13397]) Refs #2695. Merges -r13256:13396 from svn head to mythtv-channel-scanner

[danielk]

(In [13401]) Refs #2695. Refs #2053. Adds facility for saving newly scanned channels to the DB before processing them for DB insertion. This allows for debugging scans that don't resemble the developer's setup and also should allow for interactive processing in the frontend of automated channel scans in the future.

[danielk]

(In [13444]) Refs #2695. Major reorg of channel scanner code.

This moves most of the nitty gritty channel scanner stuff into libs/libmythtv/channelscan.

This also eliminates a bunch of code which is not needed anymore.

[danielk]

(In [14288]) Refs #3326. Merges r13396:14286 from trunk to mythtv-channel-scanner.

[danielk]

(In [14613]) Refs #2695. Merges r14286:14612 from trunk to mythtv-channel-scanner

[danielk]

(In [14614]) Refs #2695. Fixes a couple differences between trunk and mythtv-channel-scanner which shouldn't be there.

[danielk]

(In [14752]) Refs #2695. Merges r14612:14749 from trunk to mythtv-channel-scanner

[danielk]

(In [14910]) Refs #2695 Merges r14855:14908 from svn trunk to mythtv-channel-scan.

[Janne Grunau]

(In [14928]) Refs #2695. Merges r14908:14926 from trunk to mythtv-channel-scan

[danielk]

(In [17055]) Refs #2695. Partial merge to mythtv-channel-scan from trunk r14926:16500

[danielk]

(In [17056]) Refs #2695. Partial merge to mythtv-channel-scan from trunk r16000:16500

[danielk]

(In [17057]) Refs #2695. Partial merge to mythtv-channel-scan from trunk r16500:17056

[danielk]

(In [17103]) Refs #2695. Fixes some things missed in the merge from from trunk r14926:17056

[danielk]

(In [17104]) Refs #2695. Finshes merge of files from trunk r14926:17056

[danielk]

(In [17113]) Refs #2695. Some tweaks to get channel scan branch running again under Qt4.

[danielk]

(In [17115]) Fixes #2909. Refs #2695. Changes signal monitor code to use callbacks rather than signals/slots.

[danielk]

(In [17116]) Refs #2695. merges [17115] from trunk to channel scan branch.

[danielk]

(In [17118]) Refs #2695. Merges 17056:17117 from trunk to channel scan branch.

[danielk]

(In [17120]) Refs #2695. Refs #5228. Merges [17119] from trunk to channel scan branch.

[danielk]

(In [17173]) Refs #2695. Merges r17120:17172 from trunk to channel scanner branch.

[danielk]

(In [17182]) Refs #2695. Merges db update from channel scanner branch to trunk.

[danielk]

(In [17185]) Refs #2695. Add DVB and mux support to channel importer.

[danielk]

(In [17191]) Refs #2695. Adds human intervention for naming conflicts to command line ui for channel scanner.

[danielk]

(In [17192]) Refs #2695. Allow cardid & inputname to be specified on the command line for CLI channel scanner.

[danielk]

(In [17601]) Refs #2695. Merges r17172:17300 from trunk to channel scanner branch.

[danielk]

(In [17605]) Refs #2695. Merges r17300:17400 from trunk to channel scanner branch.

[danielk]

(In [17606]) Refs #2695. Merges r17500:17605 from trunk to channel scanner branch.

[danielk]

(In [17607]) Refs #2695. Applies changes in trunk to channel scanner branch that were missed in earlier merges.

[danielk]

(In [17621]) Refs #2695. Merges dbchannelinfo.{h,cpp} from channel scanner branch to head. This moves some channel db info classes from channelutil.{h,cpp} to the new files and adds one more for the channel scanner.

[danielk]

(In [17622]) Refs #2695. Adds a ScanDTVTransport class with a little additional information for channel scans.

[danielk]

(In [17623]) Refs #2695. Adds operator< for transports so we can sort them.

[danielk]

(In [17624]) Refs #2695. Merges some enhancements to the mpeg classes from channel scanner branch to trunk. Mostly this allows more information to be cleaned from the stream data classes for the channel scanner.

[danielk]

(In [17625]) Refs #2695. Adds function to channelutil.h which determines if a channum conflicts with existing channel numbers.

[danielk]

(In [17626]) Refs #2695. Merges CardUtil enhancements from channel scanner branch to trunk.

[danielk]

(In [17627]) Refs #2695. Adds channelscan subdir to libs/libmythtv for channel scanning stuff.

[danielk]

(In [17628]) Refs #2695. Adds channelscan files from channel scan branch to trunk; these are not compiled in yet.

[danielk]

(In [17640]) Refs #2695. Add UI for importing previous channel scans.

[danielk]

(In [17641]) Refs #2695. Adds channelscan verbose flag for channel scanner.

[danielk]

(In [17643]) Refs #2695. Reverts part of [17641] and changes the channelscan verbose flag to use some unused bits in the 32-bit mask that Nigel pointed out.

[danielk]

(In [17644]) Refs #2695. Fixes bug in CardUtil::GetInputID() introduced in [17626].

[danielk]

(In [17776]) Refs #2695. Merges r17605:17771 from head to mythtv-channel-scanner.

[Stuart Auchterlonie]

Log of --verbose channelscan,siparser,channel,record from the branch doing a full scan

[Stuart Auchterlonie]

Log of setup with --verbose siparser,channel,record against head 17783

[Stuart Auchterlonie]

Channel table after scanning with the branch

[Stuart Auchterlonie]

channel table after scanning with head 17783

[Stuart Auchterlonie]

Regressions noted in the channel scanner branch against head.

• UK LCNs are not picked up nor used.
• UK default authority descriptors are not retrieved from the SDT anymore
• Masses of duplicated channels. Total found = 264(branch) vs 82(head)
• Found some SCTE channels apparently.
• Takes *way* longer to run in the branch.

It appears that it's sitting on each frequency until it hits a timeout while in head it finds all the tables it needs and then proceeds.

Are we hitting a fallback case for broken providers even though we aren't broken?

Stuart

[Stuart Auchterlonie]

Forgot to mention, i've attached logs of both head & branch performing the same full scan.

I've also attached the resulting channel table for each run.

Stuart

[danielk]

(In [17795]) Refs #2695. Merges r17771:17794 from trunk to mythtv-channel-scanner.

[danielk]

(In [17943]) Refs #2695. Merges r17794:17942 from trunk to mythtv-channel-scanner.

[Loki]

Is it possible while working on the channelscanner to include a option to import channels without MythTV checking if the channels are correct? I'm basicly suggesting a "raw import" function besides the existing "checked imported function".

I'm asking because currently MythTV is checking the imported DVB channels following the DVB standard stricly and with this it's not possible to import correct channels when your cableprovider is using a single network id to broadcast channel information.

Of course it would be great if the import function can be fixed to make automaticly scanning work as well, but this is propably a lot more work. I can setup up a test machine if needed for this.

I'm currently adding new chapters about DVB-s/c/t to the official en wiki based manual. Can you let me know if any of this will be build/fixed? I like to include it in the manual. (you can post it on my personal page or discuss it on the myth-dev mailing list). At this moment i'm importing the channel information directly into the database, but this is not something i'm willing to suggest in a manual.

Related topics:

• Ticket 3640
• ​http://www.gossamer-threads.com/lists/mythtv/dev/339511

[danielk]

Loki, with the new scanner it is really just a matter of writing the UI for it, it keeps all the channel information throughout, including the channels in the channels.conf it did not find any additional info for.

[danielk]

(In [19421]) Refs #2695. Merges r17942:18000 from trunk to mythtv-channel-scan

[danielk]

(In [19422]) Refs #2695. Merges r18000:19416 from trunk to mythtv-channel-scan

[danielk]

(In [19423]) Refs #2695. Merges r19416:19422 from trunk to mythtv-channel-scan

[danielk]

(In [19892]) Refs #2695. Merges r19422:19887 from trunk to mythtv-channel-scan

[danielk]

(In [19899]) Refs #2695. Allow start and end transports to be specified for ATSC channel scans.

[danielk]

(In [19904]) Refs #2695. Fixes imports of existing scans. These were not being handled correctly because the tuner type was not set in ScanDTVTransport on import. Also adds some error reporting to dtv transport parsing.

[danielk]

(In [19905]) Refs #2695. Small cleanup of UI for channel scanning.

[danielk]

(In [19907]) Refs #2695. Adds toString() output for UKChannelListDescriptor.

[danielk]

(In [19930]) Refs #2695. Don't open display unless we need it in mythtv-setup

[danielk]

(In [19931]) Refs #2695. Extends command line scanner to work with dvbt.

[danielk]

(In [19932]) Refs #2695. Add freqid string to DVB-T frequency tables, this makes following the scan a little easier, has no functional effect.

[danielk]

(In [19933]) Refs #2695. Reintroduce uk channel number descriptor parsing to DVB channel scanner.

[danielk]

(In [19934]) Refs #2695. Tweaks to printing out channel and uk chan num desc handling.

[danielk]

(In [19936]) Refs #2695. Merge duplicates from channel scanning.

[danielk]

(In [19938]) Refs #2695. Fix TVCT typo caught by Stuart A.

[danielk]

UK scanning patch

[danielk]

(In [19941]) Refs #2695. Misc. fixes for DVB-T scanning in the new channel scanner.

[danielk]

(In [19942]) Refs #2695. Disable verbose messages early in command line parsing for --scan options, this allows these to be enabled later using -v/--verbose if debugging output is desired.

[Janne Grunau]

(In [19945]) Makes frequency tables useable for DVB-C and adds one for the KDG network

probably useable elsewhere. Refs #2695

[Janne Grunau]

(In [19946]) Adds missing argument to verbose string. Refs #2695

[danielk]

(In [19947]) Refs #2695. Make scanning status messages print only on changes in state if verbose output is enabled. Also limit length of status message to 80 columns so that it is less likely to cause a terminal to wrap the text.

[Janne Grunau]

(In [19948]) fix frequency tables for germany. Refs #2695

german DVB-T uses SFN instead of frequency offsets add more channels for the DVB-C frequency tables and fix the names

[Janne Grunau]

(In [19949]) modify TestNextProgramEncryption to avoid a endless loop. Refs #2695

[danielk]

(In [20290]) Refs #2695. Merges r19887:20288 from trunk to mythtv-channel-scan

[danielk]

(In [20410]) Refs #2695. Merge channel scanning verbose flag from channel scanning branch to trunk.

[danielk]

(In [20411]) Refs #2695. Various fixes from channel scanner branch. (This does not merge in the new scanner.)

[danielk]

patch to merge channel-scan branch to trunk

[danielk]

I've attached a patch that merges the channel scanning branch to trunk. After some re-testing I plan to apply it and delete the channel scanning branch so that the remaining issues can be tackled more quickly in trunk. If you see any show-stoppers please speak up...

[Stuart Auchterlonie]

(In [20415]) Refs #2695. Whitespace only cleanup of dvbtables.cpp

[Stuart Auchterlonie]

(In [20416]) Refs #2695. Adds extra debug info to NetworkInformationTable. Prints out the table section number, last section number and IsCurrent flag.

[Stuart Auchterlonie]

(In [20417]) Refs #5562, Refs #2695. Adds the Bouquet Table parsing class for BAT support.

[Stuart Auchterlonie]

(In [20418]) Refs #5562, Refs #2695. Bouquet Table plumbing. Adds the plumbing to start collecting BAT tables as they are found in the broadcast stream.

[Stuart Auchterlonie]

(In [20419]) Refs #5562, Refs #2695. Whitespace only cleanup of dvbdescriptors.cpp

[Stuart Auchterlonie]

(In [20420]) Refs #5562, Refs #2695. A bunch of dvbdescriptor updates.

Creates a ServiceDescriptorMapping class to provide a single point to convert service descriptors.

Make ServiceDescriptor use the ServiceDescriptorMapping class.

Make the BouquetNameDescriptor toString useful.

Enhance the CountryAvailabilityDescriptor to print out country codes.

Finish off ServiceListDescriptor toString. Uses ServiceDescriptorMapping class.

Adds DefaultAuthorityDescriptor.

Add plumbing to print out DefaultAuthorityDescriptor, BouquetNameDescriptor, CountryAvailabilityDescriptor, ServiceListDescriptor when present in the broadcast stream.

[Stuart Auchterlonie]

I'm just noting these for the record, in case i don't get to them before the branch is merged.

Regressions to previous versions of the channel scanner.

UK LCNs (Logical Channel Numbers) The LCN isn't found correctly if the transport being scanned is in the 2nd received part of a multi part NIT. The _received_ is important here. If you get part 2/2 first and it contains your mux then it'll work, it'll also work if you get part 1/2 first and it contains your mux. If it's in the part received 2nd, it doesn't work.

UK Default Authorities. The Default authorities are no longer picked up from the SDT. Technically they are picked up, they just don't make it to the database. However #5562 is about extending the default authority support so it can be retrieved from the BAT, so i'll address the lack of default authority in the database while working on that ticket.

Stuart

[Stuart Auchterlonie]

(In [20422]) Refs #2695. Corrects HasCachedAllNIT(). The section counter in the dvbtables starts at 0 rather than at 1.

Plus a one line whitespace change.

[Stuart Auchterlonie]

Replying to stuarta:

(In [20422]) Refs #2695. Corrects HasCachedAllNIT(). The section counter in the dvbtables starts at 0 rather than at 1.

This fixes the UK LCN problem mentioned previously...

Stuart

[Stuart Auchterlonie]

(In [20423]) Refs #2695. Don't emit ATSCStreamData::HandleTables(): Unknown table messages for tables that we know are DVB tables. Cleans up the debugging output significantly for DVB users.

[danielk]

(In [20427]) Refs #2695. Merges channel scanning branch to trunk.

[danielk]

(In [20428]) Refs #2695. Remove channel scan branch.

[satfan39]

Since the merge, the current patch for dvb-s2 ( ticket 5882 ) does not apply anylonger. This is quite an issue for those using dvb-s2. Would it be possible to obtain a new updated patch for dvb-s2

[satfan39]

The following files were "deleted" as part of the merge:

/libs/libmythtv/scanwizardhelpers.cpp /libs/libmythtv/scanwizardhelpers.h /libs/libmythtv/scanwizardscanner.cpp /libs/libmythtv/scanwizardscanner.h /libs/libmythtv/siscan.cpp /libs/libmythtv/siscan.h

It is probaly a stupid question, buyt I need to know where did the corresponding code in these 6 files go after the merge ? In other words, what are the file names that include the corresponding code after the merge. Unless I am mistaken, it is not very clear from the ticket. This info is needed for future developement and toa daot ticket 5882 ( s2api) after the merge.

[Dibblah]

Trac is not a support forum, especially not for legally dubious external patch sets.

[Stuart Auchterlonie]

(In [20652]) Refs #2695, Refs #5562. Adds default_authority column to channelscan_channel.

Default Authority's can be channel specific or mplex specific.

[Janne Grunau]

(In [21549]) revert [21040] except channelscan/channelscanner.cpp part. Refs #2695

no need to play tricks with the timeout since UpdateChannelInfo can now correctly signal that the transport is finished.

[Janne Grunau]

(In [21553]) tune only for offest == 0 by multiplex if set. Refs #2695

prevents scanning by the same multiplex data from DB multiple times if no tables are found

[danielk]

(In [21640]) Refs #2695. Move free to air only setting and replace radio services setting with more general service requirements setting. This also drops support for these being specified in the input connection table where it is ambiguous since multiple connections can be made to the same lineup and multiple scans can result in a lineup inconsistent with these settings even if the lineup is only connected to one input.

A later commit will add channel deletion prompting using these settings as a guide.

[danielk]

(In [21643]) Refs #2695. Don't attempt to bind vars removed in [21640], spams console.

[danielk]

(In [21644]) Refs #2695. Change scan progress popup to just another pane so we can see the log as we progress through the scan.

[Janne Grunau]

(In [21673]) fix channel insertion after [21644], Refs #2695

[danielk]

(In [21675]) Fixes #6638. Refs #2695. Allow user to delete stale channels and transports after a channel scan.

Note: This always asks whether to delete channels, we should probably not ask unless someone has done a full scan.

[Janne Grunau]

(In [21691]) add option to set off air channels invisble. Refs #2695

[Janne Grunau]

(In [21692]) No questions necessary if no unused transport is found. Refs #2695

[danielk]

Problems remain in channel scanning but the original problems identified in this task ticket should be fixed now. Please open new tickets for any scanning problems not described in other existing tickets.