Author: paranoidmike

A Product Manager sharing experiences on & of the job.

Silverlight training, note to self: adding non-default Assemblies for additional controls

Published on by paranoidmike1 Comment

Problem: In my Silverlight training class, we were instructed to try using a TabControl in a newspaper Silverlight app. I followed part of the instructions to add the Assembly as an xmlns reference in the page code, but I couldn’t get the sucker to compile.

Here’s the line of code I added to the page:

xmlns:my=”clr-namespace:System.Windows.Controls;assembly=System.Windows.Controls”

It was giving me this error:

The type or namespace name ‘TabControl’ does not exist in the namespace ‘System.Windows.Controls’ (are you missing an assembly reference?)

Solution: (Big forehead slap) Every single time I try to get back into .NET code, I keep having to learn this ridiculously, embarrassingly simple lesson: for whatever reason, when I add a reference to a .NET assembly in my code, I also have to add a reference to the References tree in the Solution Explorer:

  • In the Solution Explorer, browse to the Project where this code is failing
  • Right-click the References folder, choose Add Reference, and add System.Windows.Controls from the .NET tab

Silverlight training, note to self: blank page

Published on by paranoidmike3 Comments

Problem: Every time I try to debug a new project that’s derived from training materials, I end up with a blank page.

When I “View Source” on the debugging page, all I see is this:

Solution: need to set the default page in the Web project:

  • In Solution Explorer, browse the Project suffixed with “.Web”
  • Right-click on the .aspx page you wish to launch by default, and choose “Set As Start Page”

Lenovo T61? High CPU usage? Look for bad drivers (…spewing hardware interrupts)

Published on by paranoidmike3 Comments

For a couple of months, I noticed some pretty unusual CPU utilization on my Lenovo T61 Thinkpad (running Windows XP SP3).  I run Process Explorer (ProcExp) in the background all day, to keep an eye out for unusual system performance behaviour, and lately I’ve noticed that almost an entire “core” on my Intel dual-core system is consumed by something running in Kernel space.

notificationAreaI have ProcExp configured to display the CPU meter in the “systray” (aka notification area), and it’s really easy to know when there’s a kernel-consumer on your system – that’s when the CPU utilization graph will show a sizeable proportion of the line graph in red, rather than its more typical green (for User space consumers).

procexp Pop open ProcExp under this condition, and if you’ve spent as much time watching these behaviour profiles as I have, and you’ll be as surprised as I was to find the “Interrupts” process (really, just an abstract aggregate) reporting CPU utilization in the 30’s or 40’s.

 

Summary of the solution: bad drivers, needed to be updated.  In my case it was a network driver that had just been updated by the vendor; for others the culprits seem to be video drivers.  I’m sure other failures exist, but these are the two most common.

Step 1: Device Manager

When hardware and/or drivers are acting up like this, check to make sure your drivers are loading properly in Windows:

  • Open the Device Manager control panel applet (e.g. run “devmgmt.msc”)
  • Look at the list to see if any categories are expanded
  • In every expanded category, examine the details of any device that has a yellow exclamation mark (indicating there is some problem with the driver).  If at all possible, resolve the issue that’s been identified.

Until all driver issues have been resolved, any other suggestions you follow below could be moot.  If you really can’t fix the problem(s) that Device Manager is presenting, but want to try other approaches, then there are a couple of tactical workarounds you may wish to try:

  1. Install a newer (or older) version of the driver, to see if it’s just the current driver version that’s having problems.
  2. Disable the device temporarily, and make sure you are still seeing the excessive hardware interrupts being generated for a sustained period.  (If not, then perhaps that was the offending driver, and perhaps you’ll be able to run with the device disabled while the vendor addresses the issue in a later version of the driver.  Lucky you.)

Step 2: KernRate

Microsoft has an interesting little package, developed for use on Windows XP, to dig into a little detail on the Kernel-related activity on the system.  It’s not exactly a wizard I’d send to my mom, but it’s helped at least narrow the potential issues to track down for me.

You can download KernRate from here.

2.1 Setup & Config for a valid KernRate result

  • Install Windows Debugging Tools
  • Install KernRate
  • Configure valid symbols PATH variable – see this thread for details
    • Create a new directory C:\Symbols
    • Create a new environment variable by clicking on: Start menu > Control Panel > System > Advanced > Environment Variables > click New under “user variables for …”, then type:
      • Variable name: _NT_SYMBOL_PATH
      • Variable value:
  • Copy DBGHELP.DLL and SYMSRV.DLL from the install location for Windows Debugging Tools (usually C:\Program Files\Debugging Tools for Windows (x86)) to the install location for KernRate (usually C:\Program Files\KrView\Kernrates)

2.2 Use KernRate to profile kernel activity

Once it’s installed and configured correctly, you can just fire up Kernrate_i386_xp.exe, let it run for a few minutes (during the period when the Kernel activity is not what you’d expect), and then hit [Ctrl]-C to get the results of the Kernrate run.

Here’s a typical output from my system during normal operating conditions  (NOTE: for every Kernrate output, I’ve left out the entries from the final table that have no significant “hits” i.e. no significant volume of kernel events):

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe
/==============================\

\==============================/
Date: 2009/06/09   Time: 13:50:38
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
Starting to collect profile data

***> Press ctrl-c to finish collecting profile data
===> Finished Collecting Data, Starting to Process Results

————Overall Summary:————–

P0     K 0:00:11.296 (17.7%)  U 0:00:06.171 ( 9.7%)  I 0:00:46.328 (72.6%)  DPC
0:00:00.078 ( 0.1%)  Interrupt 0:00:00.031 ( 0.0%)
       Interrupts= 76929, Interrupt Rate= 1206/sec.

P1     K 0:00:12.328 (19.3%)  U 0:00:07.718 (12.1%)  I 0:00:43.750 (68.6%)  DPC
0:00:00.109 ( 0.2%)  Interrupt 0:00:00.531 ( 0.8%)
       Interrupts= 15687, Interrupt Rate= 246/sec.

TOTAL  K 0:00:23.625 (18.5%)  U 0:00:13.890 (10.9%)  I 0:01:30.078 (70.6%)  DPC
0:00:00.187 ( 0.1%)  Interrupt 0:00:00.562 ( 0.4%)
       Total Interrupts= 92616, Total Interrupt Rate= 1452/sec.

Total Profile Time = 63796 msec

                                       BytesStart          BytesStop         Byt
esDiff.
    Available Physical Memory   ,       184655872,       180600832,        -4055
040
    Available Pagefile(s)       ,      1075990528,      1069088768,        -6901
760
    Available Virtual           ,      2132107264,      2131058688,        -1048
576
    Available Extended Virtual  ,               0,               0,
  0

                                  Total      Avg. Rate
    Context Switches     ,       949736,         14887/sec.
    System Calls         ,      2754324,         43173/sec.
    Page Faults          ,       799583,         12533/sec.
    I/O Read Operations  ,         5382,         84/sec.
    I/O Write Operations ,         4657,         73/sec.
    I/O Other Operations ,        92064,         1443/sec.
    I/O Read Bytes       ,     15853022,         2946/ I/O
    I/O Write Bytes      ,      1968451,         423/ I/O
    I/O Other Bytes      ,      9179346,         100/ I/O

—————————–

Results for Kernel Mode:
—————————–

OutputResults: KernelModuleCount = 229
Percentage in the following table is based on the Total Hits for the Kernel

Time   11604 hits, 25000 events per hit ——–
Module                                Hits   msec  %Total  Events/Sec
ntkrnlpa                               9041      63796    77 %     3542933
hal                                    1155      63796     9 %      452614
win32k                                  604      63796     5 %      236691
mfehidk                                 179      63796     1 %       70145
intelppm                                163      63796     1 %       63875
igxpmp32                                112      63796     0 %       43889
HIPK                                     98      63796     0 %       38403 
……

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

And here’s the output from my system during one of these CPU over-consumption cases:

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe
/==============================\

\==============================/
Date: 2009/06/10   Time:  8:41:26
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
Starting to collect profile data

***> Press ctrl-c to finish collecting profile data
===> Finished Collecting Data, Starting to Process Results

————Overall Summary:————–

P0     K 0:02:26.906 (80.8%)  U 0:00:21.640 (11.9%)  I 0:00:13.328 ( 7.3%)  DPC
0:00:01.078 ( 0.6%)  Interrupt 0:01:18.406 (43.1%)
       Interrupts= 3919032, Interrupt Rate= 21548/sec.

P1     K 0:02:14.000 (73.7%)  U 0:00:32.703 (18.0%)  I 0:00:15.171 ( 8.3%)  DPC
0:00:00.640 ( 0.4%)  Interrupt 0:01:02.890 (34.6%)
       Interrupts= 3837550, Interrupt Rate= 21100/sec.

TOTAL  K 0:04:40.906 (77.2%)  U 0:00:54.343 (14.9%)  I 0:00:28.500 ( 7.8%)  DPC
0:00:01.718 ( 0.5%)  Interrupt 0:02:21.296 (38.8%)
       Total Interrupts= 7756582, Total Interrupt Rate= 42648/sec.

Total Profile Time = 181875 msec

                                       BytesStart          BytesStop         By
esDiff.
    Available Physical Memory   ,       564551680,       571240448,         668
768
    Available Pagefile(s)       ,      2194202624,      2233167872,        3896
248
    Available Virtual           ,      2132402176,      2131353600,        -104
576
    Available Extended Virtual  ,               0,               0,
  0

                                  Total      Avg. Rate
    Context Switches     ,      1764882,         9704/sec.
    System Calls         ,     14821228,         81491/sec.
    Page Faults          ,       188084,         1034/sec.
    I/O Read Operations  ,        31768,         175/sec.
    I/O Write Operations ,        27985,         154/sec.
    I/O Other Operations ,       321733,         1769/sec.
    I/O Read Bytes       ,     24755975,         779/ I/O
    I/O Write Bytes      ,     14122756,         505/ I/O
    I/O Other Bytes      ,     31150904,         97/ I/O

—————————–
Results for Kernel Mode:
—————————–

OutputResults: KernelModuleCount = 227
Percentage in the following table is based on the Total Hits for the Kernel

Time   62240 hits, 25000 events per hit ——–
Module                                Hits   msec  %Total  Events/Sec
ntkrnlpa                              40159     181875    64 %     5520137
hal                                   10990     181875    17 %     1510652
igxpmp32                               2381     181875     3 %      327285
HECI                                   2170     181875     3 %      298281
win32k                                 1495     181875     2 %      205498
intelppm                               1335     181875     2 %      183505
mfehidk                                 931     181875     1 %      127972
HIPK                                    660     181875     1 %       90721
Ntfs                                    361     181875     0 %       49621
usbuhci                                 285     181875     0 %       39175
PGPwded                                 223     181875     0 %       30652

……

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ============================== 

 

2.3 KernRate detailed profiling on individual modules

So what I’m seeing is (like many other folks on the ‘Net) that the NTKRNLPA module is consuming the lions’ share of the CPU (being consumed by Kernel-space threads).  Following some of the threads on the Sysinternals message boards, I dug into the details of what images were being loaded by NTKRNLPA during these kernel events:

2.3.1 NTKRNLPA

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe -s 5 -z ntkrnlpa
/==============================\

\==============================/
Date: 2009/06/10   Time:  8:49:42
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe -s 5 -z ntkrnlpa

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
CallBack: Finished Attempt to Load symbols for 804d7000 \WINDOWS\system32\ntkrnlpa.exe

Starting to collect profile data

Will collect profile data for 5 seconds
===> Finished Collecting Data, Starting to Process Results

————Overall Summary:————–

P0     K 0:00:01.984 (39.7%)  U 0:00:00.406 ( 8.1%)  I 0:00:02.609 (52.2%)  DPC
0:00:00.031 ( 0.6%)  Interrupt 0:00:01.421 (28.4%)
       Interrupts= 97650, Interrupt Rate= 19530/sec.

P1     K 0:00:02.125 (42.5%)  U 0:00:00.453 ( 9.1%)  I 0:00:02.421 (48.4%)  DPC
0:00:00.062 ( 1.3%)  Interrupt 0:00:01.406 (28.1%)
       Interrupts= 95410, Interrupt Rate= 19082/sec.

TOTAL  K 0:00:04.109 (41.1%)  U 0:00:00.859 ( 8.6%)  I 0:00:05.031 (50.3%)  DPC
0:00:00.093 ( 0.9%)  Interrupt 0:00:02.828 (28.3%)
       Total Interrupts= 193060, Total Interrupt Rate= 38612/sec.

Total Profile Time = 5000 msec

                                       BytesStart          BytesStop         Byt
esDiff.
    Available Physical Memory   ,       563130368,       560410624,        -2719
744
    Available Pagefile(s)       ,      2230300672,      2226749440,        -3551
232
    Available Virtual           ,      2132402176,      2104180736,       -28221
440
    Available Extended Virtual  ,               0,               0,
  0

                                  Total      Avg. Rate
    Context Switches     ,        73005,         14601/sec.
    System Calls         ,       214633,         42927/sec.
    Page Faults          ,         3814,         763/sec.
    I/O Read Operations  ,         1175,         235/sec.
    I/O Write Operations ,          433,         87/sec.
    I/O Other Operations ,         4569,         914/sec.
    I/O Read Bytes       ,       776604,         661/ I/O
    I/O Write Bytes      ,       129278,         299/ I/O
    I/O Other Bytes      ,       575190,         126/ I/O

—————————–
Results for Kernel Mode:
—————————–

OutputResults: KernelModuleCount = 227
Percentage in the following table is based on the Total Hits for the Kernel

Time   1762 hits, 25000 events per hit ——–
Module                                Hits   msec  %Total  Events/Sec
ntkrnlpa                               1307       4984    74 %     6555979
hal                                     246       4984    13 %     1233948
HECI                                     44       4984     2 %      220706
igxpmp32                                 38       4984     2 %      190609
intelppm                                 36       4984     2 %      180577
win32k                                   33       5000     1 %      165000
mfehidk                                  15       5000     0 %       75000
HIPK                                     12       5000     0 %       60000
……

===> Processing Zoomed Module ntkrnlpa.exe…

—– Zoomed module ntkrnlpa.exe (Bucket size = 16 bytes, Rounding Down) ——-
–
Percentage in the following table is based on the Total Hits for this Zoom Module

Time   1307 hits, 25000 events per hit ——–
Module                                Hits   msec  %Total  Events/Sec
KiDispatchInterrupt                     787       4984    46 %     3947632
READ_REGISTER_USHORT                    383       4984    22 %     1921147
READ_REGISTER_ULONG                     342       4984    20 %     1715489
READ_REGISTER_UCHAR                      41       4984     2 %      205658
KeReleaseInStackQueuedSpinLockFromDpcLevel         13       4984     0 %       65208
ZwYieldExecution                         10       4984     0 %       50160

……

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ============================== 

2.3.2 HAL

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe -v -s 5 -z hal
/==============================\

\==============================/
Date: 2009/06/10   Time: 18:25:00
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe -v -s 5 -z hal

KERNRATE: current IMAGEHLP SymOptions: UNDNAME DEBUG

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
KERNRATE: IMAGEHLP symbol search path is: .;symsrv*symsrv.dll*c:\symbols*http://
msdl.microsoft.com/download/symbols;C:\WINDOWS\System32\Drivers;C:\WINDOWS\System32;C:\WINDOWS
CallBack: Loading symbols for hal.dll…
DBGHELP: No header for hal.dll.  Searching for image on disk
DBGHELP: C:\Program Files\KrView\Kernrates\hal.dll – file not found
DBGHELP: hal.dll not found in .
DBGHELP: hal.dll not found in C:\WINDOWS\System32\Drivers
DBGHELP: C:\WINDOWS\System32\hal.dll – OK
DBGHELP: C:\WINDOWS\System32\hal.dll found
DBGHELP: .\halmacpi.pdb – file not found
DBGHELP: .\dll\halmacpi.pdb – file not found
DBGHELP: .\symbols\dll\halmacpi.pdb – file not found
CallBack: Finished Attempt to Load symbols for 806e4000 \WINDOWS\system32\hal.dll

DBGHELP: hal – public symbols
         c:\symbols\halmacpi.pdb\9875FD697ECA4BBB8A475825F6BF885E1\halmacpi.pdb
Starting to collect profile data

Will collect profile data for 5 seconds
===> Finished Collecting Data, Starting to Process Results

————Overall Summary:————–

P0     K 0:00:02.531 (50.6%)  U 0:00:00.453 ( 9.1%)  I 0:00:02.015 (40.3%)  DPC
0:00:00.000 ( 0.0%)  Interrupt 0:00:01.890 (37.8%)
       Interrupts= 119072, Interrupt Rate= 23814/sec.

P1     K 0:00:02.609 (52.2%)  U 0:00:00.656 (13.1%)  I 0:00:01.734 (34.7%)  DPC
0:00:00.031 ( 0.6%)  Interrupt 0:00:01.437 (28.7%)
       Interrupts= 114272, Interrupt Rate= 22854/sec.

TOTAL  K 0:00:05.140 (51.4%)  U 0:00:01.109 (11.1%)  I 0:00:03.750 (37.5%)  DPC
0:00:00.031 ( 0.3%)  Interrupt 0:00:03.328 (33.3%)
       Total Interrupts= 233344, Total Interrupt Rate= 46669/sec.

Total Profile Time = 5000 msec

                                       BytesStart          BytesStop         Byt
esDiff.
    Available Physical Memory   ,       111005696,       117772288,         6766
592
    Available Pagefile(s)       ,      1635250176,      1639911424,         4661
248
    Available Virtual           ,      2132402176,      2131013632,        -1388
544
    Available Extended Virtual  ,               0,               0,
  0

                                  Total      Avg. Rate
    Context Switches     ,        82262,         16452/sec.
    System Calls         ,       292287,         58457/sec.
    Page Faults          ,         6431,         1286/sec.
    I/O Read Operations  ,         1592,         318/sec.
    I/O Write Operations ,          506,         101/sec.
    I/O Other Operations ,         7687,         1537/sec.
    I/O Read Bytes       ,       344208,         216/ I/O
    I/O Write Bytes      ,       457023,         903/ I/O
    I/O Other Bytes      ,       951393,         124/ I/O

—————————–

Results for Kernel Mode:
—————————–

OutputResults: KernelModuleCount = 227
Percentage in the following table is based on the Total Hits for the Kernel

Time   1761 hits, 25000 events per hit ——–
Module                                Hits   msec  %Total  Events/Sec
ntkrnlpa                               1223       5000    69 %     6115000
hal                                     258       5000    14 %     1290000
intelppm                                 74       5000     4 %      370000
HECI                                     55       5000     3 %      275000
win32k                                   32       5000     1 %      160000
igxpmp32                                 29       5000     1 %      145000
HIPK                                     22       5000     1 %      110000
mfehidk                                  15       5000     0 %       75000
……

===> Processing Zoomed Module hal.dll…

—– Zoomed module hal.dll (Bucket size = 16 bytes, Rounding Down) ——–
Percentage in the following table is based on the Total Hits for this Zoom Module

Time   258 hits, 25000 events per hit ——–
Module                                Hits   msec  %Total  Events/Sec
READ_PORT_UCHAR                         143       5000    25 %      715000
READ_PORT_ULONG                         139       5000    24 %      695000
READ_PORT_USHORT                        139       5000    24 %      695000
HalpQueryPerformanceCounter              24       5000     4 %      120000
KeReleaseQueuedSpinLock                  21       5000     3 %      105000
KeAcquireInStackQueuedSpinLock           21       5000     3 %      105000
KfLowerIrql                               8       5000     1 %       40000
ExAcquireFastMutex                        7       5000     1 %       35000
KeGetCurrentIrql                          7       5000     1 %       35000
HalpGetCmosCenturyByte                    7       5000     1 %       35000
HalpReleaseSystemHardwareSpinLock          7       5000     1 %       35000
HalBeginSystemInterrupt                   6       5000     1 %       30000
KeAcquireQueuedSpinLock                   5       5000     0 %       25000
……

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

2.3.3 IGXPMP32

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe -v -s 5 -z igxpmp32
/==============================\

\==============================/
Date: 2009/06/10   Time: 18:25:11
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe -v -s 5 -z igxpmp32

KERNRATE: current IMAGEHLP SymOptions: UNDNAME DEBUG

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
KERNRATE: IMAGEHLP symbol search path is: .;symsrv*symsrv.dll*c:\symbols*<a href="http://
“>“>”>”>”>”>”>http://
msdl.microsoft.com/download/symbols;C:\WINDOWS\System32\Drivers;C:\WINDOWS\System32;C:\WINDOWS
CallBack: Loading symbols for igxpmp32.sys…
DBGHELP: No header for igxpmp32.sys.  Searching for image on disk
DBGHELP: C:\Program Files\KrView\Kernrates\igxpmp32.sys – file not found
DBGHELP: igxpmp32.sys not found in .
DBGHELP: C:\WINDOWS\System32\Drivers\igxpmp32.sys – OK
DBGHELP: C:\WINDOWS\System32\Drivers\igxpmp32.sys found
DBGHELP: .\igxpmp32.pdb – file not found
DBGHELP: .\sys\igxpmp32.pdb – file not found
DBGHELP: .\symbols\sys\igxpmp32.pdb – file not found
SYMSRV:  The server name or address could not be resolved
SYMSRV:  The server name or address could not be resolved
SYMSRV:  c:\symbols\igxpmp32.pdb\01C98FF4DD054B6AB11509CF9CF04F171\igxpmp32.pdb
not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/igxpmp32.pdb/01C98FF4DD054B6
AB11509CF9CF04F171/igxpmp32.pdb not found
DBGHELP: C:\WINDOWS\System32\Drivers\igxpmp32.pdb – file not found
DBGHELP: C:\WINDOWS\System32\Drivers\sys\igxpmp32.pdb – file not found
DBGHELP: C:\WINDOWS\System32\Drivers\symbols\sys\igxpmp32.pdb – file not found
DBGHELP: C:\WINDOWS\System32\igxpmp32.pdb – file not found
DBGHELP: C:\WINDOWS\System32\sys\igxpmp32.pdb – file not found
DBGHELP: C:\WINDOWS\System32\symbols\sys\igxpmp32.pdb – file not found
DBGHELP: C:\WINDOWS\igxpmp32.pdb – file not found
DBGHELP: C:\WINDOWS\sys\igxpmp32.pdb – file not found
DBGHELP: C:\WINDOWS\symbols\sys\igxpmp32.pdb – file not found
DBGHELP: C:\WINDOWS\System32\Drivers\igxpmp32.pdb – file not found
DBGHELP: D:\ccViews\autobuild1_BR-0810-1787_14.36_Snapshot\gfx_Development\dump3
2\igfx\xp\gtune\AIM3Lib\igxpmp32.pdb – file not found
CallBack: Finished Attempt to Load symbols for b942f000 \SystemRoot\system32\DRI
VERS\igxpmp32.sys

DBGHELP: igxpmp32 – export symbols
Starting to collect profile data

Will collect profile data for 5 seconds
===> Finished Collecting Data, Starting to Process Results

————Overall Summary:————–

P0     K 0:00:02.812 (56.1%)  U 0:00:00.265 ( 5.3%)  I 0:00:01.937 (38.6%)  DPC
0:00:00.000 ( 0.0%)  Interrupt 0:00:01.812 (36.1%)
       Interrupts= 114753, Interrupt Rate= 22879/sec.

P1     K 0:00:02.937 (58.8%)  U 0:00:00.562 (11.3%)  I 0:00:01.500 (30.0%)  DPC
0:00:00.062 ( 1.3%)  Interrupt 0:00:01.578 (31.6%)
       Interrupts= 109952, Interrupt Rate= 21990/sec.

TOTAL  K 0:00:05.750 (57.4%)  U 0:00:00.828 ( 8.3%)  I 0:00:03.437 (34.3%)  DPC
0:00:00.062 ( 0.6%)  Interrupt 0:00:03.390 (33.9%)
       Total Interrupts= 224705, Total Interrupt Rate= 44941/sec.

Total Profile Time = 5000 msec

                                       BytesStart          BytesStop         Byt
esDiff.
    Available Physical Memory   ,       119107584,       107962368,       -11145
216
    Available Pagefile(s)       ,      1640800256,      1629646848,       -11153
408
    Available Virtual           ,      2132402176,      2102800384,       -29601
792
    Available Extended Virtual  ,               0,               0,
  0

                                  Total      Avg. Rate
    Context Switches     ,        78959,         15792/sec.
    System Calls         ,       301466,         60293/sec.
    Page Faults          ,         8704,         1741/sec.
    I/O Read Operations  ,          711,         142/sec.
    I/O Write Operations ,          197,         39/sec.
    I/O Other Operations ,         5430,         1086/sec.
    I/O Read Bytes       ,      1212541,         1705/ I/O
    I/O Write Bytes      ,        14701,         75/ I/O
    I/O Other Bytes      ,      1486190,         274/ I/O

—————————–

Results for Kernel Mode:
—————————–

OutputResults: KernelModuleCount = 227
Percentage in the following table is based on the Total Hits for the Kernel

Time   1764 hits, 25000 events per hit ——–
Module                                Hits   msec  %Total  Events/Sec
ntkrnlpa                               1232       5000    69 %     6160000
hal                                     235       5000    13 %     1175000
intelppm                                 93       5015     5 %      463609
HECI                                     42       5015     2 %      209371
win32k                                   34       5000     1 %      170000
igxpmp32                                 32       5015     1 %      159521
mfehidk                                  23       5000     1 %      115000
HIPK                                     21       5000     1 %      105000
kmixer                                   11       5000     0 %       55000
KSecDD                                   10       5015     0 %       49850
……

===> Processing Zoomed Module igxpmp32.sys…

—– Zoomed module igxpmp32.sys (Bucket size = 16 bytes, Rounding Down) ——-
–
Percentage in the following table is based on the Total Hits for this Zoom Module

Time   32 hits, 25000 events per hit ——–
Module                                Hits   msec  %Total  Events/Sec
xc_copp_agent                            30       5015    100 %      149551

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

2.3.4 HECI

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe -v -s 5 -z heci
/==============================\

\==============================/
Date: 2009/06/10   Time: 18:48:17
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe -v -s 5 -z heci

KERNRATE: current IMAGEHLP SymOptions: UNDNAME DEBUG

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
KERNRATE: IMAGEHLP symbol search path is: .;symsrv*symsrv.dll*c:\symbols*<a href="http://
“>“>”>”>”>”>”>http://
msdl.microsoft.com/download/symbols;C:\WINDOWS\System32\Drivers;C:\WINDOWS\System32;C:\WINDOWS
CallBack: Loading symbols for HECI.sys…
DBGHELP: No header for HECI.sys.  Searching for image on disk
DBGHELP: C:\Program Files\KrView\Kernrates\HECI.sys – file not found
DBGHELP: HECI.sys not found in .
DBGHELP: C:\WINDOWS\System32\Drivers\HECI.sys – OK
DBGHELP: C:\WINDOWS\System32\Drivers\HECI.sys found
DBGHELP: .\HECI.pdb – file not found
DBGHELP: .\sys\HECI.pdb – file not found
DBGHELP: .\symbols\sys\HECI.pdb – file not found
SYMSRV:  The server name or address could not be resolved
SYMSRV:  The server name or address could not be resolved
SYMSRV:  c:\symbols\HECI.pdb\0A57FC20FA904FB8BAEC77C53EE39C821\HECI.pdb not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/HECI.pdb/0A57FC20FA904FB8BAE
C77C53EE39C821/HECI.pdb not found
DBGHELP: C:\WINDOWS\System32\Drivers\HECI.pdb – file not found
DBGHELP: C:\WINDOWS\System32\Drivers\sys\HECI.pdb – file not found
DBGHELP: C:\WINDOWS\System32\Drivers\symbols\sys\HECI.pdb – file not found
DBGHELP: C:\WINDOWS\System32\HECI.pdb – file not found
DBGHELP: C:\WINDOWS\System32\sys\HECI.pdb – file not found
DBGHELP: C:\WINDOWS\System32\symbols\sys\HECI.pdb – file not found
DBGHELP: C:\WINDOWS\HECI.pdb – file not found
DBGHELP: C:\WINDOWS\sys\HECI.pdb – file not found
DBGHELP: C:\WINDOWS\symbols\sys\HECI.pdb – file not found
DBGHELP: C:\WINDOWS\System32\Drivers\HECI.pdb – file not found
DBGHELP: d:\ccviews\autobuild1_br-0707-1049_2.6_snapshot\amt_development\sw\src\
drivers\heci\driver\objfre_wxp_x86\i386\HECI.pdb – file not found
CallBack: Finished Attempt to Load symbols for ba308000 \SystemRoot\system32\DRI
VERS\HECI.sys

DBGHELP: HECI – no symbols loaded
Starting to collect profile data

Will collect profile data for 5 seconds
===> Finished Collecting Data, Starting to Process Results

————Overall Summary:————–

P0     K 0:00:02.296 (45.9%)  U 0:00:00.343 ( 6.9%)  I 0:00:02.359 (47.2%)  DPC
0:00:00.000 ( 0.0%)  Interrupt 0:00:01.359 (27.2%)
       Interrupts= 115350, Interrupt Rate= 23070/sec.

P1     K 0:00:02.203 (44.1%)  U 0:00:00.531 (10.6%)  I 0:00:02.265 (45.3%)  DPC
0:00:00.015 ( 0.3%)  Interrupt 0:00:01.093 (21.9%)
       Interrupts= 110550, Interrupt Rate= 22110/sec.

TOTAL  K 0:00:04.500 (45.0%)  U 0:00:00.875 ( 8.8%)  I 0:00:04.625 (46.3%)  DPC
0:00:00.015 ( 0.2%)  Interrupt 0:00:02.453 (24.5%)
       Total Interrupts= 225900, Total Interrupt Rate= 45180/sec.

Total Profile Time = 5000 msec

                                       BytesStart          BytesStop         Byt
esDiff.
    Available Physical Memory   ,       116039680,       114077696,        -1961
984
    Available Pagefile(s)       ,      1642790912,      1640251392,        -2539
520
    Available Virtual           ,      2132402176,      2105364480,       -27037
696
    Available Extended Virtual  ,               0,               0,
  0

                                  Total      Avg. Rate
    Context Switches     ,        91187,         18237/sec.
    System Calls         ,       270278,         54056/sec.
    Page Faults          ,         6482,         1296/sec.
    I/O Read Operations  ,          269,         54/sec.
    I/O Write Operations ,           41,         8/sec.
    I/O Other Operations ,         5361,         1072/sec.
    I/O Read Bytes       ,      1551742,         5769/ I/O
    I/O Write Bytes      ,         2836,         69/ I/O
    I/O Other Bytes      ,      2381876,         444/ I/O

—————————–
Results for Kernel Mode:
—————————–

OutputResults: KernelModuleCount = 227
Percentage in the following table is based on the Total Hits for the Kernel

Time   1754 hits, 25000 events per hit ——–
Module                                Hits   msec  %Total  Events/Sec
ntkrnlpa                               1273       5000    72 %     6365000
hal                                     222       5000    12 %     1110000
intelppm                                 70       5000     3 %      350000
HECI                                     47       5000     2 %      235000
igxpmp32                                 34       5000     1 %      170000
win32k                                   30       5000     1 %      150000
mfehidk                                  14       5000     0 %       70000
HIPK                                     12       5000     0 %       60000
KSecDD                                   12       5000     0 %       60000
……

===> Processing Zoomed Module HECI.sys…

Symbol Enumeration failed (or no symbols found) on module HECI in CreateZoomedModuleList, Error Code= 0x000001e7
No Hits or No symbols found for module HECI.sys
================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

 

Step 3: Analysis & Research

In my case, it appears that there’s some process (or processes) that’s calling the RtlIpv6StringToAddressA kernel API an excessive number of times.  From what I’ve seen in my research, not only is this a common complaint, but it’s been reported by at least two other T61 users in the last year or so – without any definitive solution.

And what’s this RtlIpv6StringToAddressA API do exactly?  Well, it translates a string form of an IPv6 address to a more strongly-typed form of the address.  Here’s what I know (and what I don’t) about IPv6 in my T61 and Windows XP SP3:

  • The “teredo” driver hasn’t been installed in Windows XP SP3 (“teredo” is the code name used by the Microsoft Windows team that developed the IPv6 stack)
  • There are a number of third-party packages on my system, but the newest to have been introduced are:
    • the AMT 2.6-supporting software and driver (v “2.6.30.1014 built by: WinDDK”, driver date 7/12/2007)
    • the PGP whole disk encryption package
    • the Intel 4965AGN wireless driver (v12.4.0.21), installed earlier
    • the Intel PROSet/Wireless Wifi Software (v12.1.1.0), installed earlier
  • Knowing that AMT is meant to provide a remote networking interface to Intel chipset functionality, and knowing that this problem didn’t occur before the AMT & PGP installs took place, my primary suspect (as the source of these excessive IPv6-related API calls) becomes the AMT software.
  • However, I consulted with colleagues on my team that are intimately familiar with the internals of HECI.sys, and they have confirmed that it doesn’t call the RtlIpv6StringToAddressA() Win32 API
  • That prompted me to search the strings embedded in all the drivers and libraries I could find in C:\Windows\System32.  Here were the files where FINDSTR.EXE could find that API call:
    • dnsapi.dll
    • mswsock.dll
    • ntdll.dll
    • ntkrnlpa.exe
    • ntoskrnl.exe
    • Drivers\http.sys

Disabled the following Devices (which didn’t help):

  • Ricoh Memory Stick Controller
  • Ricoh SD/MMC Host Controller
  • Ricoh xD-Picture Card Controller
  • OHCI Compliant IEEE 1394 Host Controller
  • ThinkPad Modem
  • Ricoh R/RL/5C476(II) or Compatible CardBus Controller
  • SDA Standard Compliant SD Host Controller

Downloaded and installed the following driver updates:

  • bluetooth v5.5.09.6400
  • Ultranav v11.1.21.2
  • Power Manager v1.60
  • ACPI Power Management v1.53
  • Integrated camera v5.8.49004.0
  • Hotkey driver v2.20.0001

Step 4: Make sure you try later and earlier driver versions

Bottom line: I replaced the network driver on my system with one that appeared *months* after I started down this road, and I finally stopped seeing these CPU spews occur.

I was lucky – the vendor dropped a new driver version that happened to resolve this error.  Until that point, I was stuck – nothing I could do to my OS to resolve it.

And if your vendor hasn’t released a *later* driver that eliminates the issue, I’ll recommend something radical – try an *earlier* version of the driver (e.g. like these folks here).  Sometimes programmers introduce errors that didn’t exist in previous versions of their code, even if they didn’t intend to affect such a broad issue.  (Especially when they didn’t intend to.)  You might find a 6-, 12- or 24-month-old driver that still works well enough (for the time being) and doesn’t exhibit the performance drag you’re seeing.  What’s worse – an older driver that works but doesn’t have the latest features, or a driver that has the features but also craters your system performance?

Last resort: you may find that disabling the device in the OS will finally shut up a misbehaving driver (i.e. that the driver won’t load, so it won’t spawn all these excessive interrupts).  This isn’t a great long-term solution, and it’s only viable for hardware you don’t need to use (e.g. a biometric device such as fingerprint reader), but it’s one last thing to consider before junking your whole system.

Notes: related articles

Sysinternals Forums

  1. Hardware interrupts using excessive CPU
  2. Kernrate trace + RtlIpv6StringToAddressA (?)

Lenovo Forums

  1. x200 Hardware Interrupts constantly at 40% of CPU resources
  2. X200: hardware interrupts causes cpu load?
  3. T61: Problems with Hardware Interrupts when on battery
  4. Problem on S12 with flood of interrupts from Broadcom wireless-G after Windows resume

My Mac Mini HTPC saga: TV tuners

Published on by paranoidmikeLeave a comment

 

Do I really *need* a tuner?

One of the major hang-ups I had – in making the switch from Windows Media Center to a Mac Mini – was worrying whether I’d be able to plug in enough TV tuners into the Mac Mini to be able to record all the TV I usually watch.

However, after having been *without* an HTPC for almost a year, it’s amazing how little I feel like I *must* get record whatever current-run TV shows are being played.  What with the availability of almost every TV season on Netflix instant watch, or a full season on DVD, or (theoretically) a quick download via bittorrent, there’s almost no sense of urgency left to the act of watching TV, and if we happen to miss it – oh well, I’m sure I’ll be able to get it real soon. I’ve caught up on a year’s worth of The Big Bang Theory just recently, and I’m about to re-orient myself to two years’ worth of Supernatural (so I have the possibility of watching current-run episodes within the next few weeks).

All that said, I can’t quite bring myself to give up my ability to accumulate a “taped” copy of the shows that were broadcast to me, for later viewing when I have more time.  Breaking the cycle of dependence between me and the timeslot the broadcaster decides to air the show is a powerful mental leap to make, and makes me feel less like a prisoner of the networks (and their increasingly hostile attitude towards their audience).  If only to make myself feel like I have a modicum of control over my TV habits, I’ve decided to drop another $200 on an Elgato EyeTV 250 Plus tuner.

Why that brand and model?  Couple of reasons:

  1. My research into Mac-based HTPC’s and Mac-compatible tuners seems to mention the Elgato products more than any other brand.  It feels like what I saw with Hauppauge for Windows Media Center, and if my satisfaction with the Hauppauge products is any indication, the Elgato products should be good.
  2. It’s not just the hardware – apparently the EyeTV software is the best available for recording and managing playback of recordings on the Mac.

Finding all the digital channels available to be recorded with your tuner

If you’re going the OTA route, then this seems like a great way to see what all is available without a prescription (ha – I mean, subscription):

http://www.equinux.com/us/products/tubestick/tuningmap.html

If you’re stuck (like me) with wanting to get the few remaining channels that haven’t gone the full-streaming/Hulu/TV.com type of approach, and you *need* that subscription to bastards like Comcast, then your tuner has one of two uses:

  1. Analog tuner (receiving the analog output from a DTA box), in which case be prepared to buy an “IR blaster” (device that allows your HTPC to direct “change channel” commands to the IR receiver of the DTA box).  This way, you get all the channels to which you’ve subscribed – whether the cable company is encrypting those QAM signals or not.
  2. Digital tuner (receiving whichever unencrypted (aka “clear QAM”) channels your cable company hasn’t yet obfuscated with the 56-bit DES encryption that (so far) is only supportably-decryptable by the DTA and/or full cable box solutions they hook up on your behalf).  This way, you get all the channels your cable company hasn’t yet encrypted – usually including the “extended basic” channels, and probably a few others that might not be high on their list of “must-have” channels.

The EyeTV software that comes with Elgato products includes a feature called “Exhaustive Scan”, which picks up the digital channels that aren’t sent out on the typical frequencies that are publicized by Comcast.  I was able to find about 80% of the non-local channels that I normally watch (and a bunch that I’ll *never* watch, no matter *how* high-def they are).  However it took me most of the afternoon to map those channels out.

Be careful though: I don’t know why, but somehow all the configuration that I did to map those channels went up in smoke the next time I rebooted the Mini. Now maybe the system crashed before shutting down EyeTV – but is this software *really* so fragile that it doesn’t cache these settings as soon as they’re configured?  And make sure that the file that stores these settings doesn’t get wiped out by an errant half-open write operation or something?

 

Future question:

Is there a similar Plex plug-in for EyeTV as is available for Boxee?

What do you like? Your ratings in multiple places – do you want to sync them?

Published on by paranoidmike3 Comments

The modern “what do you like?” systems are driving me nuts lately.  Every time I turn around, there’s a website offering to help me socialize, and what better way to find new friends (and new recommendations for media to consume) than by accumulating a bunch of ratings for the media I’ve already consumed?

Friends (and hopefully, to some degree) passers-by can then peruse my virtual “media shelves”, see what I’ve rated and how, and either (a) get or (b) give recommendations for stuff that’s related to what they/I have already seen/read/heard and liked.

Web 1.0: Amazon Recommendations

I’ve got > 1000 ratings for graphic novels accumulated on Amazon.com – which was a great way for years for me to get recommendations on other books by the same authors, but even more importantly and gratifyingly – to get recommendations on other books by authors I hadn’t yet read.  (I’ve always assumed some sort of Bayesian analysis that results in “people who liked your 4- and 5-star books also bought/owned/rated these items”.)

[I’d further tried to accumulate lists of ratings for music (for CDs I’ve purchased), but Amazon’s interface for this wasn’t nearly as sophisticated or predictive for music as it seems to be for books, so I’ve never gotten quite the same gratifying experience for music on Amazon.com.]

Web 1.0: Netflix Recommendations

I’ve got > 1000 ratings for movies accumulated on Netflix.com – which was a great way for years for me to get recommendations on other movies I’d like that have many of the same ephemeral qualities I enjoyed in the movies I rated most highly.  (Netflix has had a highly-publicized contest – which recently wrapped up – to come up with new ways of improving user recommendations, which to me meant that they’d exhausted all the available research into Bayesian and other mathematical analysis of the huge aggregations of data on what people liked, didn’t like, watched and marked “not interested”.)

Long before I was a Netflix subscriber though, I was (and am) a diehard advocate for IMDB.com.  [Hell, I was one of the lunatic adopters in the early days back when you had to submit queries to the IMDB via email.  Yeah, imagine browsing your favourite actor’s movies [and forget about TV – that didn’t count] by submitting a cryptically-formed email message and waiting the minutes it took for their servers to generate a response.  Fred Flintstone-style browsing.]  I’ve occasionally submitted a rating for movie through that site too, though I haven’t gotten any real benefit from it (except the knowledge that I’m helping to build the geek-slanted ratings that are the killer data set available from IMDB).

Web 2.0: social + recommendations

In the past year or so, I’ve fallen deeply in love with Facebook, Twitter and all the most interesting integrations with these “social platforms”.  With these platforms have come brand-new applications that allow you to rate movies/TV/books/music/whatever and not only get some kind of recommendations back from “the system”, but also to get much more specific and immediate feedback from those of your friends (or even “friends”) who’ve also signed up to use the application.  They see what you’ve rated, then respond with comments/replies/their own ratings, and can make much more specific (and personal, though statistically less predictive) suggestions of other stuff they want you to see/read/hear.

I love these – and while I’ve experimented with a bunch of these apps, I’ve gravitated to those apps that appear to have the greatest critical mass.  Not so much because I want a horde of strangers to help me find stuff, but because I’d like to reconnect with as many friends as possible and I hope they’re also at the apps I’ve picked.

So I’ve got ratings slowly accumulating at Goodreads.com, Flixster and a couple of others.  Goodreads has a great mobile site that makes it dead-easy to post a rating “on the go” with very few excess clicks, and the Flixster iPhone app is awesome *and* easy.  And there are dozens of other great sites where lots and lots of people are accumulating lots and lots of ratings data.

Problems: stale data, incomplete data, spread-too-thin efforts

After a while, I’ve noticed I’m spread thin across multiple places where these ratings are being accumulated.  It’s an unfortunate consequence of the abundance of such great sites and platforms, that I’m finding it hard to keep my ratings “in sync” between multiple places at once.  I have good intentions – and occasionally I’ll even follow through on those good intentions. 🙂

For example, I’ve got a ton of movie ratings in Netflix, but my primary interest for “sharing” movie ratings is moving to the Flixster app – mostly because it gives me a chance to get immediate feedback from a larger group of friends who catalogue their ratings and mini-reviews there via either Facebook or the iPhone app.  However, while I’m getting immediate gratification for my posts to Flixster, it’s not doing me any good in terms of system-generated (Bayesian) recommendations for other movies I want to watch. And when I go to my Netflix queue to add movies, I sometimes forget whether I’ve seen something (since I haven’t always rated those movies I’ve seen recently).

Similarly, the primary place I currently capture my ratings for graphic novels is in GoodReads (usually via the mobile-optimized web site that I access from my iPhone).  I’m not even getting any instant feedback from friends there, nor have I found any way to use the “crowd” of GoodReads users as a source for new recommendations.  However, there’s no alternative in my Web 1.0 world: Amazon doesn’t seem to have any way to add ratings on the go.  If you’re not going through their full browser, then you’re SOL.  (The Amazon iPhone app doesn’t do squat here, and neither does the mobile browser version – it’s almost as if they don’t care whether their customers like what they bought.)

I’m now split between worlds, and I suspect the world of Facebook/social media and other Web 3.0 apps will only make this worse – there’ll be more and more sites that all want you to provide some “sticky” information, that lures in more users ‘cause there’s a “crowd” there, and yet those ratings won’t be re-usable elsewhere.

This Ratings Data Ain’t Portable, My Friends

Yeah.  Twitter might have finally gotten religion that you “own” your Tweets, and Facebook *looks* like they’re convinced that you can and should have the ability to control your personal information/updates, but good luck trying to convince the thousands of little start-up apps out there, all hoping to lure you into their little walled garden and *keep* you there.

I expect that in 5-10 years, all these systems will be able to freely consume and re-use this data – the business world will have finally gotten over thinking this is their only “value add” (and will have found some even more sexy way to separate you from your money).

However, for the forseeable future, these multiple ratings systems will continue to live as non-interoperable data islands.  That means heavy “data generators” like me will have to make some pretty dopey (and unavoidable) decisions:

  1. When you find a new, even-more-attractive place to catalogue your consumption and how much you enjoyed it, do you abandon all the invested effort you put into the last one?
  2. If you don’t want to abandon all that “legacy data”, how will you migrate it to the new system? Just devote a freakin’ weekend to the prospect of clicking like a spastic lab rat, replicating each rating from one system to the other?  Or do you go even further down the rabbit hole and learn how to export the data (if that’s supported) from one and import to another – or go completely over the cliff edge into writing yourself some web-scraping scripts that pull the data by force out of systems that don’t have a supported import/export interface (API)?
  3. And if you actually *want* to maintain a presence in more than one system – e.g. if you find some ongoing benefit in having current presence in both Netflix and Flixster?  Well gods help you then – you’re screwed into a life of regular repeated self-inflicted punishment.

I’ve really lost my mind – thinking there’s got to be some way to actually pull off (3) without feeling like a character in a Kafka novel (and no, I’ve never read Kafka, so don’t crucify me for a misspoken cultural reference).

How’s About a Ratings Sync App?

Yeah, why the hell not?  Why not just burn the next years’ worth of weekends writing an extensible framework for us to be able to download, manipulate, upload and synchronize (i.e. manage and resolve the inevitable conflicts) the ratings data?  Isn’t is just like me, to think of doing some thankless job like this, in the hopes that some morsel of thanks comes through from some other hapless geek like me?

Sure, what the hell.

In fact, I’ve invested a whole bunch of time into this harebrained notion already.  Yes, I’ve written myself a bunch of code that attempts to provide an extensible, pluggable framework in which multiple “ratings” providers could be wired in, and between which synchronization could occur.

I actually dream that one day, users like me could:

  • Fire up this app
  • Connect to one of their ratings aggregating web sites
  • Download all ratings for whatever “things” are rated on that site
  • Select another compatible ratings web site (e.g. another books-rating web site if you just download book ratings)
  • Configure a translation between the two web sites (e.g. one site rates 1-5, the other site rates 1-10, so map 1 = 2, 2 = 4, 3 = 6, 4 = 8, 5 = 10; or if you’d prefer, 1 = 1, 2 = 3, 3 = 5, 4 = 7, 5 = 9)
  • Upload the translated ratings to the second site, thus synchronizing your ratings from one site to the next

This’ll require mapping out the APIs for each ratings-aggregating site, implementing an incredible flexible and robust local schema for the data, and figuring out all the different ways that different sites identify what to the human mind is an easy-to-identify product.

[And to think, this whole idea came about as a way to figure out how to migrate the ratings from my old Netflix account (to which I’ve still got access, but only just barely – due to the grace of an old housemate) to a new one that *I* own, and from which I could actually do Netflix Instant Watch in my own personality (and with my own IW queue).  I’m seriously considering just paying them for their subscription for the rest of my life, so I never have to lose those 2956 movie ratings.]

 

Anyone out there got a better idea?

Anyone else crazy enough to want to help out with this?

Anyone out there want to see this app see the light of day?

My Mac Mini HTPC saga: software & configurations

Published on by paranoidmikeLeave a comment

Now that I’ve done the whole hardware upgrade and full, clean install of Mac OS X 10.6, I have to start all over again (this time with a little practice under my belt) on getting all the stuff assembled for a well-oiled home theatre machine.

Plex, Boxee, EyeTV, VLC, Transmission, Silverlight, MacTheRipper

Lots of software, so little attention span. 🙂

Plex

  • As my chosen media front-end, I intend to do as much as possible from here, and only veer into the other apps I’m using when necessary
  • I choose to install the following apps: Hulu, Apple Movie Trailers, South Park, Netflix, The Daily Show, Pandora, PBS, Picasa Web, Trailer Addict, YouTube

Boxee

  • I hear this is the only way to get full streaming of the CBS TV shows in a 10’ UI experience (rather than fire up Safari/Firefox and click away directly)
  • So I logged into Boxee.tv, downloaded the Mac OS X alpha of Boxee (which is simply yet another XBMC fork/port) and started looking at the “Applications” – Videos > CBS > Full Episodes shows listings for How I Met Your Mother (which I watch) but not for The Big Bang Theory (which I *slavishly* watch)

CBS: Fail.

EyeTV

  • Once I decided to definitely get a tuner (more on that in another blog article), the overwhelming number of times I’ve heard Elgato’s products recommended (not just their tuner, but their pretty great EyeTV software app to manage the tuner) was the clincher
  • The trickiest part was upgrading EyeTV from 3.0.x to 3.2 (which just came out a few days ago).  It turns out that on systems running Mac OS X 10.6, EyeTV 3.0.x won’t even launch – Apple in the infinite wisdom intentionally put a “block” in place so that it won’t start.
  • It turns out there’s an “under the covers” way to bypass such Mac OS X “blocks”, by launching the application’s actual binary file, rather than using the “user friendly shortcut” that is presented in the Applications folder (and which I, like most Mac users, seem to happily use until something prevents us from getting in the easy way).
  • I found an article at Elgato’s site that outlined the process: http://support.elgato.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=3628
  • Once I got EyeTV 3.0.x (I think it was 3.0.3 that shipped with my EyeTV 250 Plus tuner) running, I was able to use the EyeTV “Check for Updates” menu option to get the 3.2 download and be back working like a charm.

VLC Media Player

  • I remembered that VLC was the easiest way to get access to all the significant codecs, and have a nice media player in the background in case any of the rest of ‘em weren’t working out for me.
  • Downloaded, installed, and already confirmed that VLC can play VOB files that were copied directly off a DVD (for backup purposes, naturally – who wants to scratch their only copy of Robot Chicken?)

Transmission BitTorrent client

  • Ever since hearing about PeerGuardian for Windows (a piece of software that prevents your computer from connecting to “blacklisted” IP addresses – e.g. those servers setup by the RIAA, government agencies and others to track what you’re doing with your downloads and other torrent-like activity), I’ve been dreading the conversion over to my Mac Mini
  • Then somehow today, my searches through Lifehacker’s archives for “p2p file sharing ‘mac os x’” turned up an article from 2008 on Transmission – a BitTorrent client for Mac OS X and Linux.
  • It turns out that the friendly and nice folks at Transmission have integrated the PeerGuardian functionality into their app, including automatic downloads of the Bluetack blocklists
  • That’s enough for me – I’ve already got Transmission installed and slurping down some great media for later watching.

Silverlight 3.0 (for Netflix)

  • Silverlight is necessary for playing the Netflix Instant Watch streaming movies, and it doesn’t seem to get installed “in the background” by Plex, Boxee or any of the other media-front-end apps that provide a Netflix wrapper
  • Makes sense now that I think about it, but I can’t say I wasn’t disappointed by the fact that the Mac software community hasn’t made all this stuff entirely hidden from my relatively novice eyes.

MacTheRipper

Great little app for making backup copies of your DVDs onto a local hard drive.  Essential in this day and age.

Issue: Mac OS X 10.6 intercepts all Apple Remote commands

As of OS X 10.6, the operating system itself intercepts all Apple Remote commands, and cannot be overridden by application-specific configurations (as was apparently the case with Plex and probably others in the past).  Instead, hitting the Menu button on the Apple Remote will always bring up Front Row; hitting the volume buttons will always change the system-wide volume, and other irritating effects (for an HTPC user who’s using third-party HTPC-oriented software) are seen as well.

While there’s no supported, by-design way in OS X to disable this “feature”, there are some known workarounds – which I’ve implemented:

http://forums.plexapp.com/index.php?showtopic=8658

  • edited the /System/Library/LaunchAgents/com.apple.RemoteUI.plist file (after making a copy of the file in the same directory, in case I later want to revert back to the original settings)
  • deleted the /System/Library/LoginPlugins/BezelServices.loginPlugin (after making a copy of the file – ibid)

http://www.tech-recipes.com/rx/2754/os_x_edit_file_using_textedit_as_root_superuser/

I had to use this article’s cluefulness to allow me to interactively edit the RemoteUI.plist file using TextEdit.

http://support.apple.com/kb/TA25121

I also got bit by the “blank admin password” problem – wasn’t able to fire up sudo when my current account had a blank password.  So I changed it, did the sudo-enabled edit, and changed the password back.

Next Steps…

Wondering – do I really *need* a TV tuner, to enjoy the world’s best offerings of TV and other media?

My Mac Mini HTPC saga: upgrading hardware, optimizations

Published on by paranoidmike1 Comment

 

Replace RAM, Hard drive with beefier options I bought myself

I wanted to have the maximum available ‘headroom’ in this box before I committed a lot of time installing and configuring lots of software (e.g. if I wanted to run software-driven encoding, have multiple big apps running at the same time, or even to run a virtualized instance of Windows whenever I felt the ‘itch’). 

So after reading about what some of the braver souls have done under the hood, I decided that I’d purchase the lowest-end Mac Mini (with the exception of getting the fastest processor, which aren’t upgradeable AFAIK) and then purchase 4 GB of RAM & a 320 GB 7200 RPM SATA drive.

4 GB of RAM (or 8?)

While deciding on a brand of RAM is usually a decision with way too many options, on what should be but sometimes isn’t a commodity purchase, I cheated.  When browsing around Amazon.com for what kinds of prices they had available for Mac Mini’s, I noticed their “what other people purchase with this item” was consistently coming up with one package of Corsair 2 x 2GB DDR3 (PC3-8500) RAM – this one, for (at the time I purchased)

Note: I’ve done as much reading as I could about the new 64-bit capabilities unlocked by Mac OX 10.6 (Snow Leopard), and while there’s promising speculation that the Mac Mini could take 2 x 4GB RAM (i.e. 8GB total), the reality is the price of the 4GB PC3-8500 modules is astronomical compared to 2GB modules.  I’ll keep that in mind for the future, but for now the 4GB is plenty of extra capacity for now, and I can keep the extra couple hundred dollars for something else.

Which Hard Drive?

Over the decades, different hard drive manufacturers have produced drives of higher or lower quality, and I can never keep up with “which manufacturer is the king for each drive size and capacity”.  For this I trundled down to my local PC supply shop with two specifications in hand: it has to be a 2.5” drive (the bigger ‘desktop’ drives won’t fit) and it has to be a 7200 RPM drive (my reading suggests the faster drive speeds make a big difference in a Mac Mini HTPC).

The dude behind the counter was very definitive – according to him, the most reliable and best-performing 2.5” 7200 rpm drive on the market today is from Western Digital (aka the “Scorpio Black” line of drives).

If you’re looking at other brands, and you’d like to know if other Mac Mini owners have been successful upgrading with them, I found a really useful resource that could help: “Mac Drive Upgrades/Compatibility Database” at http://forums.xlr8yourmac.com/drivedb/search.drivedb.lasso

Procedure to Open the Mac and Upgrade the Components

Search Google for “Mac Mini RAM upgrade” and/or “Mac Mini hard drive upgrade” – there are many step-by-step guides available (e.g. this one), and a number of great YouTube videos that take you visually through the process. I just followed the first couple that I found and everything seemed to work fine (at least, for this “old hat” at constructing my own PCs for so many years).

For me, the only tools I needed were:

  • a putty knife or scraper blade (anything thin, metal and long enough to wedge in a few inches into the Mac Mini case)
  • a Philips (the “x” kind) screwdriver that is thin enough to get into a very narrow plastic tube (eyeglasses screwdrivers are usually good for this), and has a small/thin enough head that it can unscrew (and not strip) the screws with the very narrow “x” channel.
    • As some pointed out, magnetizing the screwdriver head is very handy for extracting those tiny little, fidgety screws – otherwise they tend to drop into the Mac Mini guts, and then you’re shaking it hard enough to get it loose (but not too hard so that you risk “damaging” some microscopic parts – not that I think there is any real danger of this with what I saw, but even an “old hat” gets worried there’s some new level of miniaturization that makes some part extremely sensitive to impact from even a 1/4-ounce screw.

Remember: Format your new Drive

Oh, and don’t forget to format the drive using Disk Utility (accessible from the top menu bar after the Mac OS X installer has asked which language you want to install).  I went through the install twice, both times wondering why the new 320GB disk wasn’t available as a selection for “where do you want to install Mac OS X”.  I figured that the installer was so smart that it recognized an unformatted disk, and would automatically format it for me.  (Soon, perhaps, but not at present.)  Embarrassing, but easily corrected – once I realized that I *had* checked all connections twice, and there was really very little that *could* prevent the drive from being recognized at the physical level.

Reinstalling Mac OS X and the rest of the bits from the two DVDs

This part was a freakin dream – so few questions, everything possible automated – it’s like Apple realizes that most people don’t care to fiddle, they just want the sucker to work.

The only extra steps I had to take were installing the optional bits (mostly utilities, but also Safari, Mail, iCal and some other useful things) from the OS X install DVD, and then installing whatever comes on that DVD labelled “”.  (Heck, it only consumed 4GB of my new 320GB drive, and until I know more about what I *don’t* need, I’m pretty comfortable taking 4GB “just in case”.)

Misc optimizations:

  • enable SMB sharing in Mac OS X, so that Plex (or Boxee) can find and scour any locally-attached external drive for photos, music and video.  (Apparently the XBMC codebase always expected to find all media on a network-shared device, since the original XBox hardware itself couldn’t retrieve or store that much media locally on the puny 20GB XBox drive.)
  • Add an entry point in the Front Row menu for Boxee – does this work for Plex as well?
  • Added link to 10-foot-UI-friendly version of my Google Reader account (via http://mymediaexperience.com/feeds-as-newspaper-in-your-living-room/)
  • Something about reconfiguring audio for Plex: “As for setting up Plex with a DTS or DD receiver. Go Configure The System -> System -> Audio Hardware from the menu. Then change the audio output from analog to digital. ” (http://www.123macmini.com/forums/viewtopic.php?t=20440)

My Mac Mini HTPC saga: filesystem selection for external media drive

Published on by paranoidmike4 Comments

 

Quest: best filesystem for my external hard drive

I have a 750 GB Seagate FreeAgent drive, where I’ve kept most of my media (photos, music, video, backups) for the past couple of years.  Until now, I was operating in an all-Windows world of my own design, and NTFS was the best (most reliable, performant, flexible) filesystem for that drive.

Now that I’ve introduced a Mac into my life, and because I’m planning to plug it into the Mac Mini for the vast majority of the time (to host media I record from or wish to display via the Mini) I’m faced with what feels like an imperfect choice:

  • FAT32 doesn’t support > 4 GB filesize, which in this day and age of recorded/downloaded TV (not to mention the potential for virtualization in my future, in case I want to keep experimenting/developing) is damned easy to exceed.
  • NTFS isn’t natively supported in Mac OS X (at least, not for writes, though a read-only NTFS driver is available).
  • Mac-only filesystems (e.g. HFS+) feel like a bit *too* much commitment to a platform that isn’t the dominant in my lifestyle yet – e.g. what if I want to unplug that drive from the Mac mini and hook it to my Windows box to do some USB 2.0-speed backups or other file transfers?  I know that most of my file transfers will probably work fine with an SMB share over the Wifi network (it’s all 802.11 g or better, in a pretty confined space), but sometimes I just want to get something done quickly.  Not to mention I’m not yet familiar with what steps I would have to take if I were to hook up my external drives (the 750GB – cleaned off now – and my backup one – FAT32), format the 750 gigger and try to copy over all my FAT32 data into reasonable facsimiles of the data I replicated off the 750 (before I repartitioned it).

There’s some fairly predictable chatter about lower performance of 3rd-party (R/W) NTFS drivers when used within Mac OS X (http://www.123macmini.com/forums/viewtopic.php?t=23243&highlight=ntfs), and I’d expect similar concerns about 3rd-party HFS+ drivers (R/W) running in Windows.  I’m not worried about booting from the external drive, so that limitation of NTFS-3g isn’t a concern for me (http://www.123macmini.com/forums/viewtopic.php?t=23057&highlight=ntfs).

However, after thinking about what my *majority* usage will be, it’s clear to me that I’ll end up leaving the drive mounted to the Mac Mini 95% of the time, so I should optimize it for that scenario and not the “just in case” fear-based scenario.  If it turns out that I’m using the drive directly attached to my Windows box *that* often, I can always use a one-time read-only HFS+ driver in Windows to get the data off the drive, then reformat with NTFS.  Getting the data off the drive isn’t a worry of mine (these drivers all seem good enough for at least the “disaster recovery – get my data off the drive *eventually*” scenarios); so I’d be an idiot not to optimize for the day-to-day performance issues I’d otherwise be facing with a non-native filesystem for the host where it’ll be plugged in 95% of the time.

Next time

  • Quest: a tuner I can “set and forget” (i.e. reliable, stable, robust performance)
  • Quest: improve the screen drawing response time in “Screen Sharing” (aka VNC)

My Mac Mini HTPC saga: Plex or Boxee as the front-end?

Published on by paranoidmike2 Comments

After opening my mind to the possibility that I could find in the Mac Mini a worthy replacement for the aging (and rotting) promise of Windows Media Center as an HTPC, it didn’t take long to find out just how far the Mac HTPC community has come.  Between the streaming media front ends like Plex & Boxee, and the first-class support from digital-TV-tuner manufacturers like Hauppauge, and the number of people who’ve blazed the trail ahead of me, it seemed like a no-brainer.

However, unlike an Apple TV (a low-powered CPU ‘appliance’ that was intended merely to download movies and TV from the iTunes store), the Mac Mini isn’t specifically designed as a home theater PC (HTPC), and doesn’t present itself to the uninitiated user as a direct complement to their TV.

That said, with not a whole lot of effort and a few add-on bits of hardware, this Mac Mini is easily the equal of the functionality of my old Windows Media Center system, and has the advantage of being a lot smaller and ridiculously quieter than a clunky monster PC.  [Outside of an extra $grand or two for a custom-designed HTPC, if you’ve got more money than brains.]

Which streaming media front end?

There are plenty of “Plex vs. Boxee” articles/forum chatter for anyone who’s looking (these are just the ones I found in my first couple of pages of Google results: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12).  There’s a lot of active development in both projects, and there’s a lot of enthusiasm for both, but there seem to be some common themes to the discussions I’ve read:

  1. Plex is more “fiddly” to setup your library of existing, local content – finding the folders and assigning the metadata.  Boxee seems to have mastered the “point it at folders and it’ll organize it into a library” approach.
  2. Boxee is more socially oriented, as in it lets other members of Boxee find out what you’ve watched (at least among the online sources – presumably it doesn’t advertise what you’re watching of your locally-stored media).
  3. One covers the BBC online media; the other covers integration of the Netflix instant watch streaming. (Or at least that’s what I recall reading.)
  4. Since they’re both based on the XBMC codebase, they’ll both benefit tremendously from future enhancements to the
  5. Boxee seems to have already skewed towards/created a culture of “cut the cable” (i.e. “help me get rid of Comcast/Time Warner stat!”), while Plex (perhaps as a Mac-only product) has a little less of the prickly attitude.
  6. They’re both generally considered inferior to Front Row from a UI/usability perspective, but neither one is ever accused of being “unusuable”.
  7. They both integrate with the Apple Remote (or if you’re a Logitech weenie, with most Harmony remotes – personally, after trying and failing for over a year to get a Harmony to operate as a set-it-and-forget-it, “universal one-touch” remote, I’ll avoid those hack-jobs like the plague).

While my girlfriend is quite adept at her Macbook, our discussions so far have led me to believe that the less fiddly/quirky this setup is on a day-to-day basis, the more comfortable she’ll be at using this (and the less I’ll have to coach her down off the ledge and into the world of computer-based home media.

Thus I’m convinced by what I’ve read to veer towards Plex (and not bother her with it until I’ve got the existing content adequately catalogued in the Plex library).

Next Steps: setting up Plex

Apparently, as one poster indicated, “The key seems to be putting Plex in Library mode. Once done, its scraper will dl the video info.”

There’s also some question whether Plex yet provides integrations with Netflix and Flickr.  I’ll have to dig through the PlexApp plug-in announcement archives.

Then I’ll be interested to dig further into the question of whether, by installing VLC media player, the codecs that it uses will be available to all media players, or just to my Mac.

Finally, I’d like to revisit the question of which desktop resolution is best suited to my 1080p HDTV – I’d tried 1920×1080, but wasn’t able to see the Mac menu bar (so e.g. I couldn’t click on the little Apple menu to reboot), so now I’m at something in the 1300-pixel-width range.