Around about the 26/3/2021 the Windows 10 MythTV add-on for Kodi stopped working. I had seen similar problem before, Kodi MythTV addon fails to load except this time it didn’t load at all whereas previously it was stuck loading at around 40%.

This time around it didn’t seem to connect to the MythTV server at all. When I tried to look at recordings, channels or recordings it just showed “PVR manager is starting up: 0%” with the spinning wheel to the right. It didn’t matter how long you waited it didn’t change.

Looking in the Kodi log, with extra add-on debugging, there was no indication of what was wrong.

This whole time MythTV server worked fine. It recorded programs fine, I could watch recordings from MythWeb. I could connect fine from Kodi on my Android phone, I could even run Myth front end, from the MythTV server, without any problems.

My MythTV setup

My setup is described in MythTV on CentOS 7 with the infrastructure looking like:

Problem dates

24/10/2022

First noticed when Shortland street latest episodes were missing. Did a restart and then MythTV didn’t even connect or at least TV didn’t show

25/03/2022

First noticed issues in the evening. I know it wasn’t a problem the night before. I did a restart of the backend MythTV server and the error persisted and I restarted the managed network switch and again the error persisted. I note that daylight savings is on 3/4/2022 at 3am. The recordings showing in listings ends on 1/4/2022 so not sure that matters – but worth noting. The fun begins …

Diagnostics

I looked at my notes from the previous similar error and tried the ideas from it, but it didn’t make any difference.

I have a daily backup of MythTV, including the database and file system. However because of my workload at work I didn’t get to look at the issue until over a week after the problem occurred so I didn’t have a backup just prior to the problem happening so I had to go back to my next most recent backup on 7/6/2020 – nearly a year old.

Rolling back the DB nearly a year resolved the problem – fantastic.

Next I went to a backup just prior to the problems, this was dated 30/March/2021. This also worked correctly. So clearly something happened in the DB between the 30th and now.

It was great the earlier backups worked, but it didn’t explain the problem and didn’t give me any clues about what to do if it came up again or what to do to prevent it. Not to mention I would have lost a years recordings.

I tried the following, individually, but none of them made any difference:

Cold start of SonicWall
Cold start of ESXi host
Stop Start of VM – as part of cold restart of ESX
Cold restart of NetGear managed switch

Kodi versions

Kodi 17, kodi-17.6-Krypton-x86.exe, which is 32 bit, works. However it doesn’t show existing recordings but live TV and recorded live TV work.

Kodi 18 and 19 do not work. Kodi 18.6 32bit also doesn’t work.

Workaround

Before you start you should ensure you have a backup strategy. The below steps should not cause long term impacts if you keep to the tables and steps listed. However it is wise to have a backup plan anyways. For myself a backup of the VM is the easiest and most full proof back – part of this is because my DB is stored on the VM. Thus a VM backup captures everything except the recording files..

It is also advisable, when testing this to do it at a time where you don’t have any critical recordings. If you need to go back to your backup you will need to remove any recordings that were made after the backup. Recordings made after the backup can easily be deleted using Windows explorer and simply deleting based on date/time.

First find out the time period where the problem is. In my case it was after the 1/4/2022

Stop MythTV backend

Stop Kodi and ensure that there aren’t any kodi.exe processes running. Use Windows task manager.

Next delete only the epg file for Kodi. In my case this file was:

C:\Users\Tony\AppData\Roaming\Kodi\userdata\Database\Epg12.db

My EPG file was “Epg12.db”, but you could find it has a different number like Epg13.db – you should delete whichever one you have. Kodi only uses one EPG file, it doesn’t work on multiples.

Run the below SQL. In my case I use DBeaver for my MariaDB work, but you could use the command line mysql, your choice,

delete from mythconverg.program
where starttime > '2022-04-01 00:00:00'
and starttime < '2022-04-05 00:00:00';
commit;

I had this problem on 24/10/2022 and was trying the workaround on 28/10/22. I found I needed to use the following query instead:

delete from mythconverg.program
where starttime > '2022-10-24 00:00:00';
commit;

Restart MythTV backend and wait until MythWeb is working and you can see listings – this proves the backend is running.

Start up Kodi – Do not use Live TV mode this may well break it again.

Problems 25/20/2021

The problem has happened again but this time I have a good backup from 25/9/21 so a month ago.

The problem happens for me in the Lounge were I have made no changes to Kodi and on my PC where I have been mucking about with Kodi. On my desktop I tried the latest Kodi, 19.3 and also 18.9. All did the same thing and hung at 0%

So I unplugged my HDHR tuners from the network so they wouldn’t update EPG. Then I rolled back to the VM and DB of 25/9/21. When I started the old VM Kodi connected straight away. Then I installed Kodi 19.3 and it also worked. They found old recordings – I only testing playing recordings with one version of Kodi but it did work.

Then I plugged in the HDHR tuners to the network and restarted the VM. Kodi now hangs at 0% when you go into “TV”. Additionally it won’t shutdown, just “Not responding” when you try to shutdown.

Next steps, stop the mythbackend, disconnect the HDHR tuners from the network. Restarted mythbackend and tried to connect with Kodi, but again it hung at 0% and would not shutdown. Then I stopped mythbackend and truncated the EPG data by running:

truncate table program;

I then restarted mythbackend. I was able to instantly go to TV recordings and play one and also it shutdown nice and normally, without hanging. This was with Kodi 19.3

Then I shutdown the VM, plugged in the HDHRs and restarted the VM. I was able to login and watch live TV and then it filled up the EPG listings

Learnings

Its not just one channel that is the problem. If I run, then Kodi works fine:

delete from mythconverg.program
where starttime > '2021-11-01 09:30:00'
and starttime < '2021-11-02 00:00:00'
and chanid in (1005,1016,1015,1013,1001,1002,1003,1007,1008,1004,1006,1009,1010);
commit;

On the other hand if I change the where clause to “and chanid in (1005,1016,1015,1013,1001,1002);” Kodi doesn’t work and equally if I change it to “and chanid in (1003,1007,1008,1004,1006,1009,1010);” it doesn’t work either. This seems to strongly suggest it is broken by multiple channels and thus multiple recordings

Testing strategy

Because I didn’t want to break anything I stopped my MythTV VM server and moved the image under /Current_Working_20210405″ and then copied the VM folder to the original location. This ensured if there were any issues with copying the VM folder it would only impact the test instance of the VM.

Secondly my recordings are on two NAS servers “NAS1” and “NAS2”. I renamed the existing “recordings” folder to “recordings – Current Recordings” and created a new folder called “recordings”.

My primary Kodi device is the lounge PC. I tried to not use this but to rather use my desktop PC to check if things were working. This is because there is quite a bit on the filesystem of Kodi and it’s associated SQLite database. If you must use your equivalent of my lounge computer then ensure you backup everything under the following folders:

C:\Users\Tony\AppData\Roaming\Kodi
C:\Program Files\Kodi

So between the above measures I was able to test ideas with the confidence my original setup was intact.

Steps
=====
Ensure all HDHR tuners are unplugged from the network

Stop Kodi and cleanup
Ensure Kodi is not running – likely need to kill in task manager
Delete the Epg *.db and TV*.db files in C:\Users\Tony\AppData\Roaming\Kodi\userdata\Database

Stop MythTV and restore DB
systemctl stop mythbackend.service
mysql -u root -p
drop database mythconverg;
create database mythconverg;
exit;
mysql -f -u mythtv -p mythconverg < /var/tmp/MySQL_DB_Dump_#0_mythconverg_MythSvr2-20211025.dmp

Run SQL scripts to remove records.

Restart Myth
systemctl start mythbackend.service
Wait for recorded programs to show in MythWeb

Restart Kodi

delete from mythconverg.program
where starttime > ‘2021-11-01 09:30:00’
and chanid in (1005,1016,1015,1013,1001,1002,1003,1007,1008,1004,1006,1009,1010);
commit;

delete from mythconverg.program
where starttime > ‘2021-11-01 09:30:00’
and starttime < ‘2021-11-02 00:00:00’
and chanid in (1005,1016,1015,1013,1001,1002,1003,1007,1008,1004,1006,1009,1010);
commit;

delete from mythconverg.record;
commit;

delete from mythconverg.program
where starttime > ‘2021-11-01 09:30:00’
and starttime < ‘2021-11-02 00:00:00’
and chanid in (1001,1002);
commit;

select * from mythconverg.program
where starttime > ‘2021-11-01 09:30:00’
and starttime < ‘2021-11-02 00:00:00’
and chanid in (1001);

delete from mythconverg.program;

select * from mythconverg.program where starttime > ‘2021-11-01 09:30:00’ ORDER BY chanid ;

select * from mythconverg.channel;

select count(*) from mythconverg.program

delete from mythconverg.program where starttime > ‘2021-11-01 09:00:00’;
commit;

select * from mythconverg.program where starttime > ‘2021-11-01 09:00:00’;

Where does “PVR manager is starting up:” come from

This is actually defined in C++ source code for Kodi, see “Resources” link at the bottom of the page:

\xbmc-Leia\addons\resource.language.en_gb\resources\strings.po

Which looks like:

#. label 'pvr starting up' for progress dialog text
#: xbmc/pvr/PVRManager.cpp
#: xbmc/pvr/windows/GUIWindowPVRBase.cpp
msgctxt "#19235"
msgid "PVR manager is starting up"
msgstr ""

This is then used on only a couple of places like

\xbmc-Leia\xbmc\pvr\windows\GUIWindowPVRBase.cpp

This doesn’t help a lot:

void CGUIWindowPVRBase::OnInitWindow(void)
{
  SetProperty("IsRadio", m_bRadio ? "true" : "");

  if (InitChannelGroup())
  {
    m_channelGroupsSelector->Initialize(this, m_bRadio);

    CGUIMediaWindow::OnInitWindow();

    // mark item as selected by channel path
    m_viewControl.SetSelectedItem(CServiceBroker::GetPVRManager().GUIActions()->GetSelectedItemPath(m_bRadio));

    // This has to be done after base class OnInitWindow to restore correct selection
    m_channelGroupsSelector->SelectChannelGroup(GetChannelGroup());
  }
  else
  {
    CGUIWindow::OnInitWindow(); // do not call CGUIMediaWindow as it will do a Refresh which in no case works in this state (no channelgroup!)
    ShowProgressDialog(g_localizeStrings.Get(19235), 0); // PVR manager is starting up
  }
}

Capturing packets to monitor requests to backend

I thought it might be really helpful to see what requests were being made as it looked like continuous requests were being made to the MythTV backend.

To do this I looked at Fiddler but it didn’t really work really well so I used Wireshark and monitored ports 6543 and 6544 using the capture filter “port 6543 or 6544.

I found that it seemed to make each api call once and did NOT make any further calls while it was hanging. It definately was not constantly calling the backend. There was also no evidence the responses were bad.

If you enable debugging you will see in the Kodi logs calls like:

2021-10-31 07:38:32.852 T:7524    DEBUG <general>: AddOnLog: pvr.mythtv: (CPPMyth)Myth::WSResponse::SendRequest: GET /Dvr/GetUpcomingList?StartIndex=0&Count=100&ShowAll=true HTTP/1.1
                                                   Host: 192.168.202.44:6544
                                                   User-Agent: libcppmyth/2.0
                                                   Connection: close
                                                   Accept: application/json
                                                   Accept-Charset: utf-8
                                                   Accept-Encoding: gzip, deflate

The resolution

It seemed clear that the problem was in the database rather than the supporting infrastructure. The question was what was wrong.

At this point I worked I was running the test VM which had a current copy of the database.

I began by truncating all the tables that I thought might be related. Then once I found the effected tables I reduced the tables to be truncated to the bare minimum to resolve the problem

Then rather than truncate I started deleting rows in the tables based on date – because it only stopped working at a particular date. This worked and I was able to reduce it to 3 tables and a handful of records

The working query is below. Obviously you would have to work out the specifics for your environment:

delete from mythconverg.record where recordid in (1157);
delete from mythconverg.program where starttime in ('2021-04-04 11:30:00');
delete from mythconverg.recorded where starttime  > ('2021-04-01 00:00:00');
commit;

For your info the rows that are being deleted look like the below. I have not been able to see any invalid or special characters that might cause a problem:

select * from mythconverg.record where recordid in (1157);
1157	4	0	08:00:00	2017-08-01	08:30:00	2017-08-01	Shortland Street		Jack meets a charmer. Michael's tempted to break the law. Nicole loses her grip.	0	0	Film	Default	10	1	6	1	3	25	Default	8	15		crid://tvnz.co.nz/1026619	crid://tvnz.co.nz/10469604		0	0	0	0	0	0	0	0	3	20:00:00	736907	0	0	0	Default	0	2021-04-05 06:57:00	2021-04-01 11:56:55	2021-04-01 12:53:54	Default	184	0	1

select * from mythconverg.program where starttime in ('2021-04-04 11:30:00') and title like '%Inside%';
1016	2021-04-04 11:30:00	2021-04-04 12:00:00	Inside Story		G Beyond the headlines to the heart of the news of the day. Al Jazeera gets the Inside Story from some of the best minds in the Middle East and beyond.		series	0	0.0	0		0	0	0	0	0	0	crid://www.aljazeera.com/15241971					crid://www.aljazeera.com/44546130	0	0	1	0	0					0	0	0

select * from mythconverg.recorded where starttime  > ('2021-04-01 00:00:00');
1002	2021-04-01 11:57:00	2021-04-01 12:50:05	Shortland Street		PG Marty struggles to find a solution. Dawn marks her territory. Zara signs out.	0	0	Movie	mythsvr2	0	0	0	1	0	Default	1157	crid://tvnz.co.nz/1031826a	crid://tvnz.co.nz/10590196		2021-04-02 01:50:05	2882862312	0.0	0		0	0	0	0	1.0	10	1002_20210401115700.ts	2021-04-01 12:00:00	2021-04-01 12:25:00	Default	Default	1	0	0	Default		1	2030	Input 1
1003	2021-04-02 04:57:00	2021-04-02 06:04:06	Newshub Live At 6pm		Newshub Live At 6pm presents comprehensive coverage of global and local news.	0	0	Entertainment	mythsvr2	0	0	0	1	0	Default	1394	crid://tv3.co.nz/68236	crid://tv3.co.nz/t1048718		2021-04-02 19:04:06	3711027128	0.0	0		0	0	0	0	1.0	0	1003_20210402045700.ts	2021-04-02 05:00:00	2021-04-02 06:00:00	Default	Default	1	0	0	Default		1	2033	Input 2
1002	2021-04-03 05:27:00	2021-04-03 07:34:05	Movie: Oddball		An eccentric chicken farmer, with the help of his granddaughter, trains his dog to protect a penguin sanctuary from fox attacks in an attempt to reunite his family and save their seaside town. (2015)	0	0	Movie	mythsvr2	0	0	0	0	0	Default	1907		crid://tvnz.co.nz/10477867		2021-04-03 20:34:05	5912729908	0.0	0		0	0	0	0	1.0	0	1002_20210403052700.ts	2021-04-03 05:30:00	2021-04-03 07:20:00	Default	Default	1	0	0	Default		1	2035	Input 1
1003	2021-04-03 04:57:00	2021-04-03 06:04:05	Newshub Live At 6pm		Newshub Live At 6pm presents comprehensive coverage of global and local news.	0	0	Entertainment	mythsvr2	0	0	0	1	0	Default	1394	crid://tv3.co.nz/68236	crid://tv3.co.nz/t1048719		2021-04-03 19:04:05	3683758480	0.0	0		0	0	0	0	1.0	0	1003_20210403045700.ts	2021-04-03 05:00:00	2021-04-03 06:00:00	Default	Default	1	0	0	Default		1	2034	Input 2
1003	2021-04-02 03:42:00	2021-04-02 04:29:05	Travels With The Bondi Vet		G New Episode. Chris goes on the celebrity trail in LA, explores a mysterious ocean outcrop in his quest to become James Bond and visits a wildlife forest in the heart of Dublin. S3 Ep1	0	0	Entertainment	mythsvr2	0	0	0	0	0	Default	357	crid://tv3.co.nz/67892	crid://tv3.co.nz/t1042608		2021-04-02 17:29:05	2866085568	0.0	0		0	0	0	0	1.0	0	1003_20210402034200.ts	2021-04-02 03:45:00	2021-04-02 04:15:00	Default	Default	1	0	0	Default		1	2031	Input 1
1008	2021-04-01 07:57:00	2021-04-01 08:40:05	Grand Designs NZ: Signature Series		PG A site with a river running through it would put most people off. But not Harlem and Nicola who attempt to build a glass-house on a bridge over the water. S5 Ep2	0	0		mythsvr2	0	0	0	0	0	Default	169	crid://tv3.co.nz/56196	crid://tv3.co.nz/t786095		2021-04-01 21:40:05	963187544	0.0	0		0	0	0	0	1.0	0	1008_20210401075700.ts	2021-04-01 07:30:00	2021-04-01 08:30:00	Default	Default	1	0	0	Default		1	2029	Input 1
1008	2021-04-02 04:42:00	2021-04-02 05:29:05	Travels With The Bondi Vet		G New Episode. Chris goes on the celebrity trail in LA, explores a mysterious ocean outcrop in his quest to become James Bond and visits a wildlife forest in the heart of Dublin. S3 Ep1	0	0	Entertainment	mythsvr2	0	0	0	0	0	Default	357	crid://tv3.co.nz/67892	crid://tv3.co.nz/t1042608		2021-04-02 18:29:05	1222027448	0.0	0		0	0	0	0	1.0	0	1008_20210402044200.ts	2021-04-02 04:45:00	2021-04-02 05:15:00	Default	Default	1	0	0	Default		1	2032	Input 1

Daylight savings – possible cause

Now I have a resolution that worked for me today, I still can’t absolutely say what caused the issue. Essentially I have to delete the most recent details in 3 tables, so it stinks of being a time related issue.

The previous similar problem seems somewhat similar and after cleaning up the tables, in that instance, things worked again. And for that case the table that was truncated was the “program” table, which is one of the 3 tables we found this time around

I documented the previous problem around November 3 last year. Daylight savings began on 27 of September 2020. Daylight savings ended April 4th 2021.

My problems this time around occurred around the 26/4/2021 with daylight savings just slightly over 8 days away on the 4/4/2021. This ties in with the EPG because the EPG holds roughly 8 days of details, so potentially EPG issues could potentially show up 8 days before the recording.